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United  States  General  Accounting  Office 
Washington,  DC  20548 


April  30,  2001 

The  Honorable  Dan  Burton 

Chairman,  Committee  on  Government  Reform 

House  of  Representatives 

Dear  Mr.  Chairman: 

In  response  to  your  requests  of  May  and  August  2000,  this  report  discusses 
our  examination  of  the  facts  surrounding  key  events  affecting  the  ability  of 
the  Clinton  Administration’s  Executive  Office  of  the  President  (EOP)  to 
preserve  certain  e-mail  messages  deemed  official  government  records.  As 
you  know,  the  Presidential  Records  Act  (PRA)  and  the  Federal  Records 
Act  (FRA)  require  the  EOP  to  preserve  official  records,  including  e-mail. 
Adequate  controls  must  be  in  place  to  ensure  that  these  records,  whether 
manual  or  electronic,  are  complete  and  maintained  in  accordance  with 
applicable  requirements.  However,  e-mail  system  malfunctions  and 
management  weaknesses  prevented  archiving  of  some  e-mail  records  for 
EOP  components,  including  the  Office  of  the  Vice  President  (OVP). 

In  our  review  of  these  events  surrounding  the  EOP’s  e-mail  systems,  you 
asked  us  to: 

•  develop  a  chronology  of  the  e-mail  malfunctions  that  included  a 
description  of  EOP  actions  taken  in  discovering  and  repairing  the 
malfunctions  and  recovering  missing  e-mail  messages; 

•  identify  officials  and  contractors  responsible  for  maintaining  the  e-mail 
system  and  correcting  the  malfunctions; 

•  determine  whether  the  OVP  implemented  adequate  practices  in 
accordance  with  applicable  laws,  regulations,  and  generally  accepted 
industry  practices  for  management  of  e-mail  records; 

•  identify  elements  contributing  to  the  estimated  cost  to  repair  the  system 
and  restore  lost  messages;  and 

•  determine  whether  EOP  officials  notified  government  officials — including 
the  Congress,  the  Justice  Department,  and  the  Office  of  Independent 
Counsel — when  the  malfunctions  were  discovered. 

To  address  these  objectives,  we  reviewed  documents  submitted  to  your 
Committee  by  the  EOP.  In  addition,  we  observed  July  through  October 
2000  U.S.  District  Court  hearings  pertaining  to  EOP  e-mail.  We  also 
submitted  written  questions  to  the  EOP,  and  reviewed  the  written 
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responses  and  additional  documentation  provided  to  us.  We  performed 
our  work  from  July  2000  through  January  2001,  in  accordance  with 
generally  accepted  government  auditing  standards;  however,  our  review 
was  limited  by  the  unavailability  of  complete  information  pertaining  to  the 
OVP’s  retention  of  hard  copy  e-mail  records,  the  EOP’s  assessment  of 
contractor  performance,  and  the  EOP’s  notification  of  government 
officials.  A  detailed  discussion  of  the  scope  and  methodology  for 
conducting  our  work  is  presented  in  appendix  I. 


Results  in  Brief 


Two  malfunctions  that  occurred  in  the  EOP  e-mail  system  prevented 
official  records  from  being  properly  recorded  in  the  Automated  Records 
Management  System  (ARMS),  which  is  a  searchable  database  of  e-mail 
records  that  was  implemented  by  the  EOP  in  1994.  The  first — an  anomaly 
with  incoming  Internet  e-mail  to  affected  users  of  the  “Mail2”  e-mail 
server — was  identified  in  late  January  1998.  In  November  1998,  in 
addressing  the  Mail2  repair,  an  EOP  contractor  introduced  a  second 
problem  that  prevented  incoming  e-mail  to  users  with  first  names  starting 
with  the  letter  D  from  being  captured  by  ARMS.  Although  these 
malfunctions  prevented  certain  e-mail  records  from  being  archived  in 
ARMS,  copies  of  these  records  were  retained  in  the  system  backup  tapes 
that  were  saved  between  November  1998  and  June  1999  to  ensure  that  the 
system  could  be  restored  after  an  interruption  in  operation.  To  ensure  that 
official  government  records  were  preserved  in  a  searchable  format,  the 
EOP  initiated  a  tape  restoration  project  in  March  2000  to  retrieve  e-mail 
records  from  available  backup  tapes  and  add  these  records  to  ARMS.  A 
detailed  chronology  of  these  events  is  presented  in  appendix  II.  Appendix 
III  contains  a  list  of  federal  officials  and  contractors  involved  in  key  events 
surrounding  maintenance  of  the  e-mail  systems  and  repair  of  the  Mail2  and 
Letter  D  malfunctions. 

The  OVP  did  not  implement  adequate  records  management  practices  to 
ensure  that  all  e-mail  records  generated  or  received  were  preserved  in 
accordance  with  applicable  law  and  best  practices.  The  OVP  initially 
followed  a  dual  approach  to  managing  its  e-mail  records  pursuant  to  the 
EOP  policy — maintaining  paper  copies  of  records  as  well  as  retaining 
backup  e-mail  tapes.  The  OVP  discontinued  this  approach  at  some  point 
after  May  1993 — in  the  mistaken  belief  that  the  e-mail  records  were  being 
archived  in  ARMS  and  that  EOP’s  Office  of  Administration  was  managing 
the  backup  tapes.  The  OVP  could  not  demonstrate  that  all  e-mail  records 
had  been  preserved  by  acceptable  methods  until  May  2000,  when  ARMS 
began  capturing  all  OVP  e-mail  records.  As  a  result,  about  600  system 


Page  2 


GAO-Ol-446  EOP  E-mail  System 


backup  tapes  needed  restoration  to  determine  if  any  non-archived  e-mail 
records  existed  on  the  tapes. 

Several  factors  contributed  to  the  expected  cost  to  restore  omitted  EOP 
e-mail  records  to  ARMS,  estimated  by  an  EOP  contractor  to  be  $11.7 
million: 

•  An  EOP  support  contractor’s  performance  of  tape  management  and 
systems  maintenance  and  documentation  activities  contributed  to  the 
increased  size  and  scope  of  the  tape  restoration  project.  Specifically,  the 
EOP’s  assessments  of  contractor  performance  revealed  that  management 
of  e-mail  backup  tapes  was  weak  and  that  maintenance  of  e-mail  systems 
and  user  accounts  needed  improvement.  In  addition,  the  automated  link 
between  the  EOP’s  e-mail  system  and  ARMS  was  not  documented  as 
required.  As  a  result,  the  EOP  could  not  identify  the  specific  tapes  within 
its  entire  tape  population  that  needed  restoration  and  could  not  readily 
understand  and  optimize  records  management  controls. 

•  The  EOP  did  not  effectively  monitor  management  of  e-mail  records.  It  did 
not  include  critical  elements  in  its  monitoring  program,  such  as 
evaluations  of  the  creation  of  records,  maintenance  and  use  of  records, 
and  records  disposition.  This  increased  the  time  needed  to  detect  the 
Mail2  and  Letter  D  malfunctions  and  added  to  the  accumulation  of  backup 
tapes  to  be  restored. 

•  Scrutiny  of  the  e-mail  malfunctions  and  the  EOP’s  tape  restoration 
practices  by  external  authorities  introduced  additional  project  tasks, 
including  contracts  for  independent  verification  and  validation  of  the 
restoration  process  and  for  additional  security  over  the  project. 

Although  the  EOP  knew  of  the  malfunctions  that  prevented  e-mail  from 
being  captured  by  ARMS  from  October  1996  through  mid-May  1999, 
officials  stated  that  they  did  not  understand  the  breadth  and  scope  of  the 
impact  of  these  malfunctions  on  the  EOP’s  response  to  subpoenas  and 
production  of  other  documents  until  February  2000.  The  EOP  claims  that 
it  notified  appropriate  investigative  bodies  at  that  time.  However,  we  were 
unable  to  obtain  sufficient  evidence  to  confirm  that  these  notifications 
were  made. 

We  requested  comments  on  a  draft  of  this  report  from  the  EOP  and  the 
personal  representatives  of  former  President  Clinton  and  former  Vice 
President  Gore.  The  Deputy  General  Counsel,  Office  of  Administration, 
EOP,  orally  provided  a  technical  clarification,  which  was  incorporated. 
The  Counsel  to  former  Vice  President  Gore  provided  written  comments 
stating  that  (1)  the  report  did  not  explicitly  state  that  we  found  no 
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evidence  of  attempts  by  OVP  staff  to  deliberately  fail  to  preserve  records, 
and  (2)  ARMS  was  a  unique  system  for  which  best  practices  did  not  apply 
and  additional  information  on  the  creation  and  use  of  this  system  was 
provided.  In  regard  to  the  statements  regarding  the  intent  of  OVP  staff,  our 
review — as  requested — focused  on  the  OVP’s  management  of  e-mail 
records.  We  did  not  assess  the  intent  of  individuals  responsible  for 
managing  these  records  and,  as  such,  made  no  conclusions  on  this  subject. 
Regarding  ARMS,  we  disagree  with  the  Office  of  the  former  Vice 
President’s  characterization  of  the  applicability  of  best  practices  to  ARMS. 
The  best  practices  cited  in  this  report  currently  exist  in  the  form  of 
National  Archives  and  Records  Administration  guidance  and  apply  to 
records  management  programs,  irrespective  of  the  type  of  system  used. 
However,  we  have  revised  the  report  to  include  the  additional  information 
on  ARMS  that  was  provided.  The  Office  of  the  former  Vice  President  also 
provided  technical  comments  that  we  have  incorporated  as  appropriate. 
The  representative  of  former  President  Clinton  provided  a  letter,  which 
stated  that  he  did  not  have  access  to  the  records  necessary  to  properly 
review  and  comment  on  its  accuracy. 


Background 


PRA,  set  forth  in  title  44  of  United  States  Code,  chapter  22,  requires  the 
President  and  Vice  President  to  adequately  record  their  official  acts, 
maintain  certain  official  records,  and  transfer  custody  of  such  records  to 
the  Archivist  of  the  United  States  upon  termination  of  their  terms  of  office. 
Pursuant  to  the  PRA,  both  the  Office  of  the  President  and  the  Office  of  the 
Vice  President  are  to  implement  records  management  controls  and  other 
necessary  actions  to  ensure  that  presidential  and  vice  presidential 
activities,  deliberations,  decisions,  and  policies  are  adequately 
documented  and  maintained.  It  also  provides  that  presidential  records 
shall  be  made  available  pursuant  to  subpoena  or  other  judicial  process  and 
to  either  House  of  Congress,  and  prohibits  destruction  of  records  without 
prior  concurrence  of  the  Archivist  and  notification  of  Congress  60  days 
prior  to  disposal.  The  Archivist  promulgates  standards  and  guidance  for 
implementation  of  PRA;  these  are  contained  in  title  36  of  the  Code  of 
Federal  Regulations  (CFR).  This  regulation  defines  criteria  for  access  to 
and  disposition  of  presidential  records. 

Federal  records  are  subject  to  the  provisions  of  FRA,  set  forth  in  title  44 
U.S.C.,  chapters  29,  31,  and  33.  FRA  requires  the  heads  of  federal  agencies 
to  make  and  preserve  records  documenting  the  official  activities  of  the 
agency.  FRA  directs  federal  agencies  to  establish  (1)  a  program  for  the 
management  of  agency  records,  (2)  effective  controls  over  the  creation, 
maintenance,  and  use  of  records,  and  (3)  safeguards  against  the  removal 
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or  loss  of  records.  The  Archivist  promulgates  standards  and  guidance  for 
implementation  of  FRA,  which  are  contained  in  title  36  CFR.  Requirements 
for  federal  records  are  more  extensive  than  are  those  for  presidential 
records,  and  specify  that  federal  agencies 

•  preserve  records  documenting  the  official  activities  of  federal  agencies  to 
protect  the  legal  rights  of  the  government  and  of  persons  directly  affected 
by  the  agency’s  activities; 

•  periodically  monitor  staff  determinations  of  the  record  status  of 
documentary  materials; 

•  implement  a  records  maintenance  program  so  that  complete  records  are 
filed  or  otherwise  identified  and  preserved  and  that  records  can  be  found 
when  needed; 

•  implement  an  agencywide  program  for  the  management  of  all  federal 
records  created,  received,  and  stored  on  electronic  media; 

•  ensure  compliance  with  federal  criteria  and  review  electronic  information 
systems  periodically  for  conformance  with  established  agency  procedures, 
standards,  and  policies; 

•  develop  and  maintain  up-to-date  documentation  about  all  electronic 
information  systems  adequate  to  specify  technical  characteristics, 
understand  the  purpose  and  function  of  the  systems,  and  ensure  timely, 
authorized  disposition  of  records; 

•  consider  specific  criteria  when  developing  procedures  for  maintenance  of 
electronic  mail  records  in  recordkeeping  systems,  including  copying-, 
records  from  the  e-mail  system  to  a  separate  recordkeeping  system, -Mand 

•  retain  records  from  e-mail  systems  in  an  off-line  storage  format.  Agencies 
that  use  electronic  formats  (such  as  optical  disk  or  magnetic  tape)  must 
maintain  the  ability  to  convert  the  records  to  the  National  Archives  and 
Records  Administration’s  required  format  and  medium  at  the  time  of 
transfer.  Agencies  that  maintain  paper  files  as  their  recordkeeping  systems 
are  required  to  print  their  electronic  mail  records  and  the  related 
transmission  and  receipt  data. 

To  ensure  the  adequacy  of  management  and  control  of  federal  information 
systems,  the  Office  of  Management  and  Budget  (OMB)  issued  circulars  A- 
123,  Management  Accountability  and  Control,  and  A-130,  Management  of 
Federal  Information  Resources.  Circular  A-123  requires  that  federal 


*A  recordkeeping  system  is  a  manual  or  automated  system  in  which  records  are  collected, 
organized,  and  categorized  to  facilitate  their  preservation,  retrieval,  use,  and  disposition. 
Because  they  do  not  have  the  features  specified  by  the  National  Archives  and  Records 
Administration,  system  backup  tapes  are  not  to  be  used  for  recordkeeping  purposes. 
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agencies  implement  adequate  internal  controls  and  management 
structures  to  ensure-the  effective  operation  of  federal  programs.  Appendix 
III  of  circular  A-130,  Security  of Federal  Automated  Information 
Resources,  requires  that  federal  agencies  periodically  test  system  controls 
meant  to  ensure  the  integrity,  confidentiality,  and  availability  of 
information  resources. 


The  EOP  Policies 
Required  Management 
of  Records 


The  Administration  issued  guidance  to  clarify  the  responsibilities  of  each 
EOP  component  regarding  presidential,  federal,  and  personal  records  in 
order  to  implement  the  provisions  of  both  PRA  and  FRA.  A  detailed  policy 
and  procedures  document  issued  by  the  EOP  in  May  1993  required  that  no 
federal  or  presidential  e-mail  records  could  be  deleted  unless  maintained 
in  an  electronic  recordkeeping  system  or  printed  and  placed  in  a  file.  In 
addition,  it  required  that  the  White  House  Records  Management  Office 
monitor  electronic  systems  to  ensure  that  correct  record  status 
determinations  had  been  made  for  presidential  anil-federal  e-mail  records. 
This  policy  was  applicable  to  all  EOP  components, Hncluding  the  OVP. 


In  May  1997,  the  White  House  replaced  the  PRA  section  of  the  1993  policy 
with  one  that  provided  a  summary  of  White  House  policies  and  guidance 
regarding  PRA.  It  required  preservation  of  all  original  presidential  records 
and  all  materials  covered  by  subpoena  or  other  such  requests.  It  also 
directed  each  staff  member  to  determine  the  record  status  of  outgoing  e- 
mail  messages  and  to  maintain  hard  copy  records  in  organized  files. 
According  to  EOP  officials,  the  policy  issued  in  1997  was  substantively  the 
same  as  the  1993  policy,  and  it  served  as  a  reminder  to  staff  of  the  policy 
originally  articulated  in  the  1993  policy  regarding  compliance  with  PRA. 


History  Surrounding 
the  EOP  E-mail 
Systems 


A  series  of  events  occurred  between  July  1994,  when  ARMS  was 
implemented,  and  March  2000,  when  the  tape  restoration  project  was 
initiated.  These  events  are  detailed  in  appendix  II.  Various  EOP  federal 
and  contractor  staff  were  involved  in  the  maintenance  and  repair  of  the 


2OMB  circular  A-130  applies  to  EOP’s  Office  of  Administration  component,  which  had 
responsibility  for  electronic  systems,  including  e-mail  and  ARMS. 

3The  EOP  consists  of  the  White  House  Office,  the  OVP,  and  other  components,  such  as 
OMB  and  the  Council  on  Environmental  Quality.  EOP  components  that  create  and  receive 
presidential  records  included  the  White  House  Office,  the  OVP,  the  National  Security 
Council,  and  the  Office  of  Policy  Development. 
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e-mail  systems  between  October  1996  and  June  1999;  they  are  listed  in 
appendix  III. 

According  to  the  EOP,  in  order  to  comply  with  the  Armstrong  ruling, ^it 
contracted  with  Information  Management  Consultants,  Incorporated,  to 
develop  the  ARMS  recordkeeping  system.  ARMS  began  archiving  e-mail 
records  in  1994.®-* 


Since  1996,  the  EOP  had  used  Lotus  Notes®  software  as  its  e-mail 
application.  During  the  former  administration,  the  Office  of  Administration 
within  the  EOP  maintained  four  Lotus  Notes  e-mail  servers,  one  remote 
server,  and  one  ARMS  interface  server,  which  transferred  e-mail  records 
from  the  e-mail  system  to  ARMS. 

E-mail  messages  designated  as  records  subject  to  either  PRA  or  FRA  were 
passed  to  ARMS  through  the  interface  between  the  two  systems.  The 
Notes-to-ARMS  interface  program  automatically  identified  unrecorded 
e-mail  messages  when  it  periodically  scanned  for  new  messages  in 
groupings  of  user  accounts  called  “views.”  User  accounts  were  distributed 
to  a  view  depending  on  the  first  letter  of  the  account  name.  Then,  the 
ARMS  interface  copied  these  new  messages  for  processing  to  ARMS  and 
marked  the  messages  as  “recorded.”  In  its  comments  on  a  draft  of  this 
report,  the  Office  of  the  former  Vice  President  stated  that  the  ARMS 
database  maintained  e-mail  records  in  a  searchable  format,  which  came  to 
facilitate  the  EOP’s  responses  to  document  requests  and  subpoenas 
sometime  much  later  in  the  administration.  A  key  feature  included  in 
ARMS  was  the  ability  to  perform  a  computerized  search  of  e-mail  using 
keyword  terms  to  search  for  responsive  materials. 

From  October  1996  through  May  1999,  two  malfunctions  occurred  in  the 
EOP  e-mail  system  that  prevented  incoming  Internet  e-mail  from  being 
properly  archived  in  ARMS.  In  late  January  1998,  an  EOP  employee 
initially  identified  the  first  of  these  malfunctions — an  anomaly  with 
incoming  Internet  e-mail  to  affected  users  of  the  Mail2  e-mail  system 
server — and  documented  this  in  an  incident  report.  According  to  EOP 


4Armstronq  v.  EOP.  810  F.  Supp.  335  (1993).  The  court  in  the  Armstrong  case  held  that  the  then- 
current  EOP  records  management  practice  did  not  save  all  information  from  an  electronic  record,  and 
thus  violated  FRA. 

5The  EOP  initially  used  ARMS  with  the  e-mail  system  that  preceded  use  of  the  Lotus 
Notes®  e-mail  system. 
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officials,  efforts  were  undertaken  then  to  analyze  this  anomaly,  but  it  was 
thought  to  be  an  isolated  event.  According  to  these  officials,  the  precise 
cause  of  the  Mail2  malfunction  was  determined  in  June  1998  by  Northrop 
Grumman,  the  EOP’s  information  technology  services  support  contractor 
since  October  1997,  when  performing  other  work  on  the  Lotus  Notes  e- 
mail  system  interface  to  the  ARMS  records  management  database.  The 
malfunction  was  caused  by  improper  user  account  configuration  in  which 
the  server  name  “Mail2”  was  spelled  “MAIL2.”  Because  the  ARMS  interface 
program  did  not  recognize  the  upper  case  spelling  of  the  mail  server  name, 
it  was  unable  to  locate  and  capture  new  incoming  e-mail  messages  for 
these  user  accounts.  This  malfunction  was  subsequently  repaired  in 
November  1998. 

Also  in  November  1998,  in  addressing  the  Mail2  repair,  a  second  problem 
was  introduced  by  Northrop  Grumman  staff  that  prevented  incoming 
Internet  e-mail  to  users  with  first  names  starting  with  the  letter  D  from 
being  captured  by  ARMS.  It  involved  omission  of  the  letter  D  from  the 
Lotus  Notes  view  (described  above),  causing  the  ARMS  scanner  to  skip 
those  accounts  beginning  with  the  letter  D  and  not  capture  incoming  e- 
mail  to  the  affected  user  accounts.  This  malfunction  was  discovered  in 
April  1999  and  repaired  in  May  1999.  Appendix  IV  presents  a  time  line  of 
key  events  pertaining  to  the  discovery  and  repair  of  the  Mail2  and  Letter  D 
malfunctions. 

The  EOP  e-mail  systems  were  periodically  backed  up,  or  copied,  to  tape 
media  for  use  in  the  event  of  system  failure  and  subsequent  need  to 
restore  the  system  to  full  operation.  In  addition,  Northrop  Grumman 
followed  a  standard  industry  practice  of  economical  tape  recycling.  This 
involved  retaining  the  tapes  for  3  weeks  and  then  recycling  them — 
resulting  in  either  overwriting  the  data  on  the  tapes  or  destroying  the  used 
tape  and  replacing  it  with  a  new  tape.  Because  the  backup  tapes  contained 
copies  of  e-mail  not  processed  to  the  ARMS  records  management 
database,  the  EOP  stopped  its  normal  practice  of  tape  recycling  from  June 

1998,  when  the  cause  of  the  Mail2  malfunction  was  documented,  until  June 

1999,  after  the  Letter  D  malfunction  was  repaired. 

In  March  2000,  the  EOP  implemented  the  tape  restoration  project,  for 
which  it  hired  Enterprise  Computing  Solutions  Technology,  Incorporated 
(ECS)  to  recover  the  lost  e-mail  messages  and  restore  the  messages  to  the 
ARMS  records  management  database.  The  EOP  hired  another  contractor, 
Vistronix,  to  oversee  the  work  of  ECS.  In  July  2000,  Vistronix  estimated 
that  the  project  would  cost  about  $11.7  million  for  restoration  of  about 
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6,000  tapes.  Table  1  presents  a  breakdown  of  the  $11.7  million  budgeted 
cost.  Appendix  V  presents  further  details  on  this. 


Table  1:  Summary  of  Budgeted  Tape  Restoration  Costs. 

Budgeted  cost 
element 

Description  of  cost  element 

Estimated 

cost 

ECS  contract 

Restoration  of  e-mail  messages  not  recorded  in 
ARMS 

$5,125,829 

Vistronix  contract 

Independent  validation  and  verification  of  the 
tape  restoration  project 

1,993,819 

AAC  Associates 
contract 

Advice  on  recovery  of  OVP  e-mail 

900,000 

Special  services 

Forensic  analysis  and  recovery  of  broken  or 
unreadable  tapes 

2,092,400 

Direct  costs 

Additional  hardware,  software  licenses,  and 
12,000  tapes  for  two  sets  of  tape  copies 

498,480 

Other 

Project  overrun  contingency 

1,098,891 

Total 

$11,709,419 

Source:  Tape  restoration  project  cost  estimate  report  prepared  by  Vistronix,  Incorporated,  for  the 
EOP  in  July  2000. 


The  OVP  Did  Not 
Ensure  Adequate 
Preservation  of  E-mail 
Records 


According  to  the  EOP,  although  the  1993  EOP  records  management  policy 
applied  to  the  OVP,  the  OVP  did  not  implement  adequate  practices  to 
ensure  continual  preservation  of  its  e-mail  records  in  accordance  with 
PRA,  which  required  implementation  of  records  management  controls. 

The  OVP’s  internal  policy  from  1993  until  some  time  in  1998  was  twofold — 
to  rely  on  its  staff  to  retain  its  e-mail  server  backup  tapes  and  to  maintain 
files  of  hard  copy  e-mail  records.  According  to  the  EOP,  however,  from 
May  1993  until  May  2000,  the  OVP  could  not  demonstrate  that  its  practices 
for  managing  presidential  e-mail  records  provided  continual  preservation 
of  records  during  the  former  administration. 


During  a  portion  of  the  administration,  the  OVP  was  responsible  for 
operation,  maintenance,  and  records  management  of  its  own  e-mail  system 
separate  from  that  used  by  the  EOP.  In  March  1998,  responsibility  for 
operation  and  maintenance  of  the  system  transferred  to  the  Office  of 


According  to  EOP,  a  December  21,  2000,  letter  from  EOP  to  the  National  Archives  and 
Records  Administration  documented  agreement  reached  regarding  OVP’s  records 
management  approach.  However,  our  review  of  this  letter,  obtained  from  the  National 
Archives  and  Records  Administration,  found  the  letter  to  discuss  disposition  of  e-mail 
records  upon  the  presidential  transition  and  it  did  not  mention  the  OVP  records 
management  approach. 
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Administration.  However,  the  OVP  did  not  communicate  its  requirements 
for  management  of  e-mail  records  and  assumed  that  records  management 
responsibility  also  transitioned  to  the  Office  of  Administration.  According 
to  EOP,  the  Office  of  Administration  assumed  that  the  OVP  had  retained 
records  management  responsibilities  and  did  not  take  steps  to  ensure  that 
OVP  e-mail  records  were  appropriately  preserved. 

According  to  the  EOP,  “at  some  subsequent  point”  since  May  1993,  the 
OVP  discontinued  retention  of  paper  copies  of  e-mail  records  because  it 
believed  its  e-mail  to  be  captured  by  ARMS.  This  mistaken  belief  was 
based  on  the  precedent  that  ARMS  managed  e-mail  records  for  all  other 
EOP  components  since  its  implementation  in  1994.  In  its  comments  on  a 
draft  of  this  report,  the  Office  of  the  former  Vice  President  stated  that  this 
belief  was  also  based  on  the  fact  that  searches  of  ARMS  produced  OVP  e- 
mail  records  and  on  the  observation  that  some  outgoing  OVP  e-mails  were 
in  fact  being  managed  by  ARMS  (as  discussed  below).  The  EOP  estimated 
that  “by  1998,”  the  OVP  discontinued  retention  of  paper  copies  of  e-mail 
records  not  captured  by  ARMS  (but  subject  to  PRA).  Therefore,  paper 
copies  of  e-mail  records  did  not  exist  for  a  period  of  at  least  2  years  (1998 
and  1999)  and  up  to  7  years  (1993  through  1999).  In  its  comments  on  a 
draft  of  this  report,  the  Office  of  the  former  Vice  President  stated  that 
these  records  are  being  restored  as  part  of  the  tape  restoration  project.  We 
requested  access  to  files  of  any  hard  copy  e-mail  records  covering  this 
period,  but  it  was  not  provided. 

Although  the  OVP  did  not  continually  retain  paper  copies  of  e-mail  records 
in  accordance  with  the  EOP  policy,  it  did  retain  the  backup  tapes  created 
through  late  March  1998  (except  for  a  backup  malfunction  from  mid- 
February  to  mid-March  1994).  According  to  the  EOP,  the  OVP  backup 
tapes  were  for  system  recovery  and  records  archival,  but  they  were  not  in 
a  text-searchable  format. 

After  transition  of  the  e-mail  system  in  March  1998,  the  Office  of 
Administration  maintained  the  OVP  e-mail  system  in  accordance  with  its 
established  operating  practices  for  other  EOP  systems.  One  practice 
followed  was  to  recycle  system  backup  tapes  every  3  weeks.  Application 
of  this  practice  to  the  OVP  system  resulted  in  irretrievable  loss  of  the 


'According  to  the  EOP  in  June  2000,  the  OVP  tapes  were  saved  for  compliance  with  PRA.  In 
January  2001,  the  EOP  stated  that  the  OVP  did  not  routinely  search  its  backup  tapes  in 
response  to  subpoenas  and  that  only  through  the  tape  restoration  project  has  the  OVP 
demonstrated  that  the  tapes  are  recoverable. 
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electronic  form  of  e-mail  records  while  tapes  were  recycled  between 
March  1998  and  March  200(L  Another  practice  was  to  use  the  standard 
EOP  ARMS  e-mail  template4x)  create  new  OVP  e-mail  user  accounts 
between  March  1998  and  March  2000,  which  inadvertently  caused  ARMS 
to  capture  outgoing  e-mail  records  sent  by  these  users.  The  result  of  this 
practice  was  that  ARMS  captured  outgoing  e-mail  for  those  OVP  users  that 
had  the  EOP  e-mail  template  (133  of  157  OVP  users).  When  it  learned  in 
March  2000  that  ARMS  did  not  fully  preserve  OVP  e-mail  records,  the  EOP 
ordered  that  tape  recycling  be  stopped  and  initiated  corrective  action- 
According  to  the  EOP,  all  OVP  e-mail  users,  except  Senate  OVP  staff, ^ 
became  fully  recorded  in  ARMS  as  of  May  8,  2000.  However,  because  the 
OVP  did  not  ensure  the  preservation  of  e-mail  records,  in  either  paper  or 
electronic  form,  presidential  records  may  have  been  irretrievably  lost.  In 
addition,  about  600  backup  tapes  were  added  to  the  restoration  project  to 
recover  as  many  e-mail  records  as  possible. 


Several  Factors  Have 
Driven  Cost  of  E-mail 
Restoration 


Several  factors  affected  the  size  and  scope  of  the  tape  restoration  project 
and  the  related  estimated  cost  of  $11.7  million  to  restore  omitted  e-mail 
records  to  ARMS.  First,  an  EOP  support  contractor’s  performance  led  to 
increases  in  the  number  of  tapes  for  restoration  and  adding  to  the  passage 
of  time  necessary  to  diagnose  system  anomalies  and  repair  malfunctions. 
Second,  the  EOP’s  monitoring  of  e-mail  records  did  not  provide  reasonable 
assurance  that  ARMS  captured  all  required  records,  which  increased  the 
time  needed  to  detect  the  Mail2  and  Letter  D  malfunctions  and  added  to 
the  accumulation  of  backup  tapes  for  the  restoration  effort.  Finally,  tasks 
were  added  to  the  project  due  to  legal  scrutiny  of  the  e-mail  malfunctions 
and  the  EOP’s  tape  restoration  practices. 


The  EOP’s  Support 
Contractor  Did  Not 
Effectively  Perform  Tasks 
Related  to  E-mail  Systems 


In  supporting  its  information  technology  functions,  the  EOP  had  a  primary 
support  services  contract,  which  required  the  contractor  to  provide 
common  services  to  all  users  of  the  EOP  data  center.  The  data  center 
provides  general  computer  operations,  server  and  network  support,  and 
software  support.  Under  its  contract,  Northrop  Grumman  was  required  to 


The  ARMS  e-mail  template  was  configured  to  automatically  pass  to  ARMS  a  blind  copy  of 
all  e-mail  records  sent  by  an  EOP  e-mail  user. 

^Because  the  Vice  President  also  serves  as  the  President  of  the  Senate,  some  OVP  staff 
support  the  Vice  President’s  Senate  duties.  Some  Senate  OVP  staff  used  a  separate  e-mail 
system  controlled  by  the  Senate. 
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•  “manage  cartridge  tapes,  monitor  catalog  assignments,  maintain 
appropriate  tape  management  systems  (automated  on  IBMs  and  VAXs, 
manual  on  all  other  platforms),  [and]  maintain  back  up  system  tapes  in 
accordance  with  Government  standards/procedures”  as  part  of  its 
management  of  the  data  center  tape  library; 

•  develop  and  support  Notes  e-mail,  administer  Notes  systems  including 
server  hardware  and  software  configuration,  maintain  mail  backup 
systems,  resolve  hardware  and  software  problems  for  the  ARMS  interface, 
provide  technical  support  for  operation  and  maintenance  of  the  interface, 
and  develop  and  implement  quality  assurance  programs  for  this  work;  and 

•  develop  system  documentation  to  capture  the  functions,  interface,  and 
internal  control  requirements,  and  document  requirements  of  existing 
systems  that  need  updating  or  were  not  previously  documented. 

The  contract  also  included  provisions  for  EOP  oversight  and  monitoring  of 
contractor  performance.  The  EOP  established  contract  performance 
measures  and  an  associated  performance  evaluation  plan,  which  provided 
a  mechanism  for  formal  measurement  and  documentation  of  contractor 
performance  and  allocating  award  fees  and  penalties  based  on 
performance.  The  EOP  conducted  formal  award  fee  assessments  of 
contractor  performance  twice  per  year,  in  accordance  with  the  contract. 

In  monitoring  the  contractor,  the  EOP  found  weaknesses  in  the 
contractor’s  performance  of  data  center  tape  management  tasks.  The 
EOP’s  award  fee  assessment  report  from  October  1998  to  mid-April  1999 
noted  that  tape  management,  in  general,  had  been  a  weak  point  in  the 
contractor’s  performance.  Specifically,  the  report  stated  that  the 
contractor  had  not  created  an  inventory  of  the  e-mail  system  backup  tapes, 
and  that  these  tapes  continued  to  mount  in  an  uncontrolled  manner.  As  a 
result,  the  EOP  rated  the  contractor’s  performance  as  marginal  and 
withheld  about  $10,400  in  award  fees  for  this  and  other  data  center  tasks. 

The  EOP’s  subsequent  award  fee  assessment  report  for  the  period  of  mid- 
April  to  September  1999  stated  that  there  continued  to  be  a  lack  of 
effective  tape  management  in  the  data  center  and  rated  the  contractor’s 
performance  as  unsatisfactory.  We  were  not  able  to  determine  what,  if 


10 According  to  FRA,  manual  external  labels  or  automated  tape  management  systems  for 
tapes  storing  electronic  records  must  provide  unique  identification,  including  the  name  of 
the  unit  responsible  for  the  data  and  system  title.  Other  identifying  information  such  as  file 
title(s),  dates  of  creation,  volume  serial  number,  and  software  dependency  must  be 
maintained  but  not  necessarily  attached  to  the  tape. 
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any,  award  fee  was  withheld  because  the  EOP  did  not  provide  us  with 
complete  award  fee  documentation.  The  problems  with  tape  management 
were  not  subsequently  resolved.  In  March  2000,  the  EOP  tape  restoration 
project  manager  stated  in  a  memorandum  that  e-mail  backup  tapes 
needing  restoration  could  not  be  identified  and  that  all  tapes  that  had  been 
retained  would  need  to  be  examined  to  determine  their  contents. 

The  EOP  also  documented  deficiencies  in  the  contractor’s  performance  of 
e-mail  system  and  user  account  maintenance.  According  to  the  EOP,  the 
contractor  had  not  implemented  satisfactory  quality  controls  to  prevent  or 
detect  mistakes  in  user  account  and  system  configurations.  As  a  result,  the 
EOP  did  not  detect  the  Letter  D  malfunction  for  5  months  and  the  Mail2 
malfunction  for  almost  2  years.  For  example: 

•  The  EOP’s  award  fee  assessment  report  for  October  1998  through  mid- 
April  1999  stated  that  the  configuration  error  that  caused  the  Letter  D 
malfunction  “should  not  have  occurred  and  [it]  could  have  been  fixed  on 
the  spot  had  [the  contractor  implemented]  proper  testing  procedures  and 
quality  assurance  measures  been  in  place.”  As  a  result,  the  EOP  rated  the 
contractor’s  performance  in  this  area  as  unsatisfactory  and  withheld 
$20,832  in  award  fees  for  this  and  other  system  life  cycle  tasks. 

•  The  error  in  configuration  of  user  accounts  that  caused  the  Mail2 
malfunction — entering  “MAIL2”  instead  of  “Mail2” — continued  even  after  it 
was  discovered  in  June  1998.  In  April  1999,  the  EOP  found  that  this 
configuration  error  had  not  only  occurred  continually  on  the  Mail2  server 
since  October  1996,  but  also  on  another  EOP  e-mail  server.  The  EOP 
initiated  a  review  of  controls  to  detect  such  configuration  errors  in  May 
1999. 

Finally,  up-to-date  system  documentation  for  the  e-mail  system  interface 
to  ARMS  was  not  maintained,  as  required  by  the  contract  and  under  FRA. 
Accurate  system  documentation  and  availability  of  source  code  facilitate 
an  understanding  of  how  a  system  functions,  which  enables  the  system 
owner  and  system  administrator  to  diagnose  system  anomalies — such  as 
those  experienced  with  the  Mail2  and  Letter  D  malfunctions — and 
optimize  system  controls.  However,  although  a  subcontractor  to  a  former 
EOP  contractor  documented  the  system  when  it  developed  the  interface 
between  the  Lotus  Notes  and  ARMS  systems,  the  Office  of  Administration 
staff  found  that  both  the  system  documentation  and  the  programming 
source  code  were  missing  in  1997.  Until  the  source  code  was  located  in 
1998  and  the  EOP  contractor  documented  the  system  functions  in  1999, 
the  EOP  was  unable  to  implement  changes  to  the  code  or  develop 
automated  programs  to  monitor  the  effective  functioning  of  the  interface. 
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These  shortcomings  in  tape  management,  system  and  account 
maintenance,  and  system  documentation  contributed  to  the  time 
necessary  for  the  EOP  to  detect  the  malfunctions  and  ultimately  hindered 
the  EOP’s  identification  of  the  e-mail  backup  tapes  requiring  restoration, 
increasing  the  quantity  of  tapes  that  would  have  to  be  examined  in  the 
project.  From  the  time  between  identification  and  repair  of  the 
malfunctions — for  Mail2,  June  to  November  1998;  for  Letter  D,  April  to 
May  1999 — and  restoration  of  the  tapes  that  began  in  March  2000,  the  EOP 
continued  to  accumulate  backup  tapes.  This  resulted  in  raising  the  total 
number  of  tapes  to  be  restored  from  788  in  December  1998  to  about  4,500 
in  January  2001  (comprised  of  3,900  Mail2  and  Letter  D  tapes  and  600  OVP 
tapes). 


While'  the  EOP’s  1993  policy  required  that  the  White  House  Records 
Management  Office  monitor  electronic  systems  containing  presidential 
and  federal  e-mail  records,  the  EOP’s  records  management  program  did 
not  provide  reasonable  assurance  that  adequate  controls  were  in  place  and 
functioning  to  monitor  the  preservation  of  such  records.  Such  a 
monitoring  program  should  include  evaluations  of  the  creation  of  records, 
maintenance  and  use  of  records,  and  records  disposition  to  determine 
compliance  with  established  agency  and  federal  recordkeeping 
requirements. 

Pursuant  to  its  policy,  the  EOP  implemented  a  monitoring  program  in  July 
1994  to  identify  systemic  problems  with  records  determinations  made  by 
EOP  e-mail  users.  The  EOP’s  monitoring  program  applied,  however,  only 
to  federal  records  and  did  not  cover  monitoring  of  presidential  records  as 
required  by  the  EOP  policy.  In  addition,  the  monitoring  program  did  not 
include  tests  to  ensure  the  completeness  of  federal  and  presidential  e-mail 
records  captured  by  ARMS  that  are  created  by  incoming  e-mail  messages 
to  the  EOP  e-mail  users.  Controls  were  specifically  needed  over  these 
incoming  messages  because  the  e-mail  system  allowed  the  EOP  users  to 
delete  incoming  messages  before  they  were  automatically  copied  to 
ARMS.  This  ability,  coupled  with  the  EOP’s  normal  practice  of  recycling 
backup  tapes,  may  have  resulted  in  the  irretrievable  loss  of  the  electronic 
form  of  e-mail  records.  However,  the  EOP  did  not  document  or  regularly 
test  the  adequacy  of  records  management  controls  until  May  1999  after 
repair  of  the  Letter  D  malfunction,  when  the  EOP  implemented  an 
automated  e-mail  monitoring  program.  Because  controls  were  not  in  place 
until  May  1999,  the  Letter  D  malfunction  went  undetected  for  5  months, 
adding  additional  backup  tapes  to  the  size  of  the  restoration  project. 


Monitoring  Program 
Did  Not  Ensure 
Effective  Management 
of  E-mail  Records 
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Legal  Scrutiny  Added 
Restoration  Tasks 


Scrutiny  over  the  tape  restoration  project  by  the  U.S.  District  Court, ^ 
Office  of  the  Independent  Counsel,  and  the  Congress,  began  in  March 
2000,  introducing  additional  tasks.  Examples  of  such  tasks  include  use  of 
law  enforcement  imaging  software  (required  by  the  Office  of  the 
Independent  Counsel)  and  development  of  an  interim  searchable  database 
and  hiring  of  an  independent  validation  and  verification  contractor  and 
security  guard  support  to  enhance  integrity  of  tape  processing  (required 
by  the  U.S.  District  Court  and  the  Congress). 

According  to  the  EOP,  the  restoration  project  was  initiated  in  March 
2000 — not  in  1998  after  repair  of  Mail2  or  in  1999  after  repair  of  Letter 
D — because  the  EOP’s  focus  in  1999  was  on  remediation  of  its  systems  to 
ensure  continued  processing  after  the  year  2000  and  leap  day  (February 
29,  2000)  rollovers.  As  a  result,  the  quantity  of  e-mail  messages  requiring 
restoration  escalated,  drawing  the  attention  of  authorities  interested  in 
ascertaining  whether  the  EOP  had  implemented  adequate  controls.  Table  2 
lists  restoration  project  activities  estimated  at  about  $3.5  million  that  were 
introduced  because  of  this  scrutiny. 


Table  2:  Additional  Project  Activities  and  Related  Costs. 

Project  activity 

Cost  (estimated) 

Independent  verification  and  validation  contractor 

$1,993,819 

Security  (two  contracts): 

General  Services  Administration 

TW  &  Company 

(actual)  97,920 
54,894 

Development  of  interim  searchable  database 

963,400 

Tape  inventory  and  catalogue  (list)  tape  contents 

455,000 

Total 

$3,565,033 

Source:  Tape  restoration  project  cost  estimate  report  prepared  by  Vistronix,  Incorporated,  for  the 
EOP  in  July  2000,  the  EOP  summary  of  actual  costs  through  October  29,  2000,  and  discussions  with 
EOP  officials. 


nThe  tape  restoration  project  was  being  monitored  by  Judge  Lamberth  of  the  U.S.  District  Court 
incident  to  the  case  of  Alexander  v.  FBI  (CA  96-2125,  D.D.C.).  The  plaintiffs  in  that  case  had 
submitted  document  requests  to  the  EOP  in  an  attempt  to  discover  relevant  information  in  EOP’s  e- 
mail. 
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Limited  Access  to 
Information 
Prevented 
Confirmation  of  the 
EOP’s  Notification  of 
Appropriate  Officials 


The  EOP  claimed  to  have  notified  investigative  bodies  as  soon  as  it 
understood  that  document  production  might  have  been  affected  by  the  e- 
mail  malfunctions.  Although  the  cause  of  the  Mail2  e-mail  malfunction  was 
known  as  early  as  June  1998  and  Letter  D  in  April  1999,  the  White  House 
Counsel’s  Office  stated  that  it  did  not  discover  until  February  2000  that  the 
e-mail  malfunctions  “had  affected  the  integrity  of  White  House  document 
productions.”  Thus,  “the  Counsel’s  Office  did  not  perceive  a  need  to  notify 
investigative  bodies”  prior  to  that  time. 

According  to  EOP  officials,  when  the  extent  of  the  malfunctions  became 
clear  to  the  White  House  Counsel’s  Office  in  February  2000,  that  office 
“had  both  written  and  oral  communications  with  various  investigative 
bodies  regarding  the  Mail2  issue.”  The  EOP  provided  us  with  copies  of  its 
correspondence  to  the  House  Committee  on  Government  Reform  that 
range  in  time  from  March  17  to  October  30,  2000.  On  March  17,  2000,  the 
EOP  provided  the  Committee  with  a  description  of  the  e-mail  system,  the 
malfunctions,  and  their  effect  on  document  production.  A  letter  to  the 
Committee  in  June  2000  updated  the  status  of  e-mail  records  management 
for  the  Office  of  the  Vice  President.  Finally,  an  October  6,  2000,  letter  from 
the  EOP  to  the  Committee  referred  to  30  letters  from  the  EOP  to  various 
investigative  bodies,  between  March  15  and  October  3,  2000,  including  the 
Offices  of  Independent  Counsel,  congressional  committees,  and  the 
Department  of  Justice. 


We  requested  copies  of  these  30  letters,  but  we  were  not  given  copies  of 
any  of  them.  Without  complete  documentation,  we  were  unable  to  confirm 
the  EOP’s  claims  to  have  notified  officials  concerning  the  effect  of  the  e- 
mail  malfunctions  on  respective  document  productions  to  investigative 
bodies  other  than  the  Committee.  In  its  comments  on  a  draft  of  this  report, 
the  Office  of  the  former  Vice  President  stated  that  as  a  matter  of  policy, 
“the  EOP  did  not  cross-pollinate  its  communications  with  various 
investigative  bodies  as  a  means  of  maintaining  the  confidentiality  of  the 
legitimate  interests  and  investigative  priorities  of  those  investigative 
bodies.” 


Conclusions 


Computer  malfunctions,  ineffective  systems  and  records  management 
practices,  and  miscommunication  between  EOP  components  led  to  e-mail 
records  not  being  preserved  by  ARMS.  As  a  result,  the  EOP  initiated  a 
time-consuming  and  expensive  effort  to  recover  e-mail  records  that  had 
not  been  effectively  managed. 
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The  management  control  weaknesses  have  implications  for  the  current 
administration  that  inherited  the  core  e-mail  systems  and  records 
management  structures  discussed  in  this  report.  Accordingly,  we  met  with 
the  current  Special  Assistant  to  the  President  and  Director  of  the  Office  of 
Administration  and  interested  EOP  staff,  to  provide  suggestions  for 
establishing  effective  controls  in  the  records  management  and  e-mail 
system  support  functions.  These  officials  stated  that  actions  are  underway 
to  improve  the  management  of  electronic  records  within  EOP.  These 
efforts  included  developing  and  updating  policies  for  federal  and 
presidential  records,  defining  procedures  for  electronic  records,  and 
establishing  a  chief  information  officer  position. 


Agency  Comments 
and  Our  Evaluation 


In  written  comments  on  a  draft  of  this  report,  which  are  reprinted  in 
appendix  VI,  the  Counsel  to  former  Vice  President  Gore  had  two  general 
points.  First,  he  stated  that  the  report  did  not  explicitly  state  that  we  found 
no  evidence  of  attempts  by  OVP  staff  to  deliberately  fail  to  preserve 
records.  In  our  report,  our  objective  was  to  examine  the  facts  surrounding 
the  e-mail  systems.  Specifically,  we  were  requested  to  assess  the  adequacy 
of  management  controls  over  EOP’s  records  management  system, 
particularly  as  they  related  to  the  numerous  e-mail  system  malfunctions 
experienced  by  EOP  since  1996.  That  assessment  entailed  the  examination 
of  management  policies,  programs,  and  practices  as  they  related  to 
effective  records  management  controls  and  to  the  requirements  of  the  PRA 
and  FRA.  Our  review  was  not  directed  at  assessing  the  intent  of 
individuals  responsible  for  preserving  records  and  providing  responsive 
records  to  investigative  bodies.  As  such,  we  can  offer  no  conclusions  on 
this  matter. 


Second,  the  Office  of  the  former  Vice  President  stated  that  ARMS  was  a 
unique  system  for  which  best  practices  did  not  apply  and  provided 
additional  information  on  the  creation  and  use  of  this  system  that  was  not 
reflected  in  the  draft  report.  In  response,  we  have  revised  the  report  to 
reflect  this  additional  information.  However,  we  disagree  with  the  Office 
of  the  former  Vice  President’s  characterization  of  the  applicability  of  best 
practices  to  ARMS.  Best  practices  for  management  of  records  currently 
exist  in  the  form  of  National  Archives  and  Records  Administration 
guidance  contained  in  title  36  CFR  and  apply  to  records  management 
programs  irrespective  of  the  type  of  system  involved.  The  majority  of  this 
guidance  is  prescriptive  only  for  federal  records;  however,  this  guidance 
applies  to  ARMS  because  ARMS  archives  federal  records.  Also,  the 
guidance  does  not  explicitly  preclude  application  of  these  practices  in  the 
management  of  presidential/vice  presidential  records.  Although  the 
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evolution  of  ARMS  may  continue,  the  stability  of  generally  accepted 
records  management  best  practices  establishes  a  control  foundation  for 
any  system,  whether  manual  or  automated. 

Comments  from  the  Office  of  the  former  Vice  President  are  reprinted  in 
appendix  VI.  In  addition,  this  appendix  contains  our  detailed  responses  to 
numerous  technical  comments  that  were  provided. 

In  his  comments  on  the  draft  report,  the  representative  of  former 
President  Clinton  stated  that  he  had  reviewed  the  draft  but  did  not  have 
access  to  the  records  necessary  to  properly  review  and  comment  on  its 
accuracy.  He  urged  us  to  incorporate  comments  that  we  received  from 
EOP  during  the  course  of  our  review,  which  we  have  done  as  appropriate. 

The  Deputy  General  Counsel,  Office  of  Administration,  EOP,  orally 
provided  a  technical  clarification,  stating  that  the  May  1997  policy  that 
replaced  the  PRA  section  of  the  1993  policy  was  issued  by  the  White 
House  and  not  EOP,  which  we  incorporated  as  appropriate. 


We  are  sending  copies  of  this  report  to  the  Honorable  Henry  Waxman, 
Ranking  Minority  Member;  the  Honorable  Hector  Irastorza,  the  Deputy 
Assistant  to  the  President  for  Management  and  Administration,  the  White 
House;  Bruce  R.  Lindsey,  the  personal  representative  of  former  President 
Clinton;  Andrew  M.  Wright,  Counsel  to  former  Vice  President  Gore;  and 
interested  congressional  committees.  Copies  will  then  be  available  on  our 


Web  site  at  lwww.gao.gov 


If  you  or  your  office  have  any  questions  about  this  report,  please  contact 
me  at  (202)  512-6240  or  by  e-mail  at  KoontzL@gao.gov.  Nancy 
DeFrancesco,  Michael  Fruitman,  Linda  Lambert,  and  Charles  Roney  were 
major  contributors  to  this  report. 

Sincerely  yours, 

Jjru,  Jr  i 

Linda  D.  Koontz 

Director,  Information  Management  Issues 


Enclosures 
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Appendix  I:  Scope  and  Methodology 


To  obtain  information  regarding  events  and  to  identify  individuals  key  to 
the  Executive  Office  of  the  President’s  (EOP)  e-mail  systems  and  the 
malfunctions  experienced,  we 

•  reviewed  about  10,000  pages  of  documentation  submitted  to  the  House 
Committee  on  Government  Reform  by  the  EOP  and  its  contractor 
Northrop  Grumman; 

•  reviewed  159  hearing  exhibits  prepared  by  the  Committee  from  the  above 
documentation,  and  pertinent  pieces  of  correspondence  and  other  related 
documentation  provided  by  the  Committee; 

•  obtained  copies  of  the  EOP’s  task  order  OA8004  dated  September  30,  1997, 
with  Northrop  Grumman,  and  letter  contract  OA20C002  dated  March  29, 
2000,  with  Enterprise  Computing  Solutions  Technologies,  Incorporated 
(ECS); 

•  obtained  transcripts  of  a  hearing  pertaining  to  the  timing  of  the  restoration 
and  production  of  EOP  e-mail,  held  by  the  U.S.  District  Court  beginning 
July  13,  2000,  and  observed  the  hearing  sessions  between  July  31  and 
October  3,  2000;  and 

•  obtained  transcripts  of  hearings  pertaining  the  EOP  e-mail  systems,  held 
by  the  House  Committee  on  Government  Reform  on  March  23,  March  30, 
May  3,  and  May  4,  2000. 

To  obtain  additional  information  on  the  e-mail  malfunctions  and  Office  of 
the  Vice  President’s  e-mail  records  procedures  that  were  not  addressed  by 
the  Committee  documents  or  hearing  transcripts,  the  EOP  stipulated  that 
we  submit  written  questions.  We  submitted  seven  sets  of  questions  and 
reviewed  the  written  responses  and  additional  documentation  provided, 
including  two  additional  tape  restoration  project  contracts:  Vistronix, 
Incorporated,  letter  contract  CI20C03  dated  May  8,  2000;  and  TW  and 
Company  contract  DC21C01  dated  November  9,  2000.  We  also  met  with 
EOP  officials  to  discuss  specific  issues  pertaining  to  the  restoration  of  the 
e-mail. 

To  determine  costs  for  the  project,  we  obtained  copies  of  paid  ECS 
invoices  covering  the  period  May  5  through  September  5,  2000.  We  also 
obtained  a  copy  of  the  July  28,  2000,  Vistronix  report  that  estimates  project 
costs  and  compared  this  estimate  to  updated  information  regarding 
project  status  and  actual  costs  incurred  for  the  project  through  October 
29,  2000,  that  we  obtained  from  the  EOP. 

Our  review  of  cost  issues  was  limited  by  the  unavailability  of  complete  and 
reliable  information.  Specifically,  we  were  not  given  access  to  OVP  files  of 
hard  copy  e-mail  records,  which  impaired  a  full  assessment  of  the  OVP’s 
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practices  to  preserve  e-mail  records.  In  addition,  we  were  not  provided 
complete  documentation  of  contract  performance  assessments  and  related 
consequences  for  fiscal  years  1998,  1999,  and  2000;  thus,  we  were  unable 
to  determine  the  extent  to  which  the  EOP  documented  contractor 
performance  weaknesses  and  withheld  award  fees.  Finally,  we  were  not 
provided  with  copies  of  letters  from  the  EOP  to  investigative  bodies 
necessary  for  us  to  confirm  that  the  EOP  notified  these  entities  of  the  e- 
mail  malfunctions. 

We  performed  our  work  from  July  2000  through  January  2001,  in 
accordance  with  generally  accepted  government  auditing  standards. 
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Appendix  II:  Chronology  of  E-mail 
Malfunctions 

Reference8 

documents 

Date 

Description  of  event8 

Cross-reference 
to  other 
documents 
listed  in  this 
table8 

Comments8 

E5346  through 
E5347,  E5353 
through  E5354 

07/94  and 
10/94 

E-mail  from  Nell  Doering  to  various  staff 
including  Daniel  Barry,  James  Wright,  and 
others  establishing  an  e-mail  monitoring 
program  that  was  implemented  07/25/94.  The 
Presidential  and  Office  of  the  Vice  President 
(OVP)  Records  Monitoring  Plan  dated  1 0/28/94 
that  was  attached  provided  a  quarterly 
monitoring  schedule  through  fiscal  year  (FY) 
1995. 

This  monitoring  program 
required  that  EOP 
agencies  review  a  sample 
of  record  and  nonrecord 
e-mail  messages  to 
ensure  that  proper  record 
designation  was  applied 
by  e-mail  users. 

E0595  through 
E0608 

03/07/95 

A  document  entitled  Automated  Records 
Management  System  (ARMS)  for  Lotus  Notes 
Mail  provided  a  technical  description  of  the 
Notes/ARMS  interface  functionality.  The 
document  stated  that  the  interface  copies  all 
internal  and  external  e-mail  and  receipts  to  the 
ARMS  records  management  database.  White 
House  Executive  Office  of  the  President  (EOP) 

E5346  through 
E5347  and  E5353 
through  E5354 

Nonrecords  were  also 
captured  temporarily  by 
ARMS  for  sampling  under 
EOP’s  records  monitoring 
program  to  detect  and 
correct  improper 
designation  of  e-mail 
records. 

e-mail  users  may  elect  to  remove  the  record 
designation  of  an  outgoing  mail  message, 
which  they  created,  by  selecting  a  nonrecord 
command  button.  This  option  causes  the 
message  to  not  be  captured  as  an  official 
federal  or  presidential  record  to  be  archived  in 
the  ARMS  records  management  database. 


El  1 15 

08/29/96 

The  Mail2  server  was  created  with  the  server 
name  of  Mail2  and  User  ID  of  Mail2/EOP. 

E0564 

E0564  through 

10/29/96 

E-mail  stating  that  a  Planning  Research 

Exhibit  125  and 

PRC  was  the  predecessor 

E0566 

Corporation  (PRC)  contract  employee  installed 
the  Mail2  server,  as  well  as  performed  initial 
configuration  of  the  Notes  server.  The 
Notes/ARMS  interface  Name  and  Address 

Book  (NAB)  was  configured  with  five  views. 

document  El  1 15 

contractor  to  Northrop 
Grumman  (NG)  at  the 

White  House. 

E2309  through 

02/25/97 

Series  of  e-mail  between  Daniel  Barry  and 

Based  on  available 

E2310 

and 

02/26/97 

Laura  Crabtree  regarding  "...  ‘problem 
attachment’  problem”  whereby  messages  were 
deleted  manually  if  they  caused  a  problem  with 
the  Notes/ARMS  interface. 

documentation,  this 
problem  was  not  related 
to  the  Mail2  or  Letter  D 
malfunctions. 

Exhibit  17 

03/04/97 

E-mail  from  Daniel  Barry  to  John  W.  McGinnis 
asking  him  to  write  a  paragraph  regarding 
efforts  performed  to  restore  10,000+  lost  E-mail 
records.  Mr.  McGinnis’  contribution  would  finish 
up  Mr.  Barry’s  “memo  to  the  record.” 

The  3/23/00 
hearing  witness 
list  lists  Mr.  Barry 
as  Daniel  “Tony” 
Barry.  He  was 
referred  to  as 

Tony  in  many  e- 
mails  and  other 
documents. 

Based  on  available 
documentation,  this 
problem  was  not  related 
to  the  Mail2  or  Letter  D 
malfunctions. 
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Malfunctions 


Reference8 

documents 

E231 6  through 
E2323 


E6001  through 
E6004 


E6547  through 
E6558 


Date 

03/06/97 


03/17/97 


03/21/97 


Cross-reference 
to  other 
documents 
listed  in  this 

Description  of  event8  table8 

Series  of  e-mails  from  Daniel  Barry  to  James  Exhibit  17 

Wright  and  others  regarding  “missing  Notes 

records.”  The  problem  occurred  in  September 

1996  and  caused  some  e-mail  to  be  sent  to  the 

ARMS  records  management  database  without 

the  message  header.  Other  e-mails  would  be 

sent  to  the  ARMS  records  management 

database  without  the  message  text.  Mr.  Barry 

stated  he  had  received  no  training  on  Notes.  He 

reported  that  to  his  knowledge,  efforts  to 

recover  these  missing  e-mail  messages  in  the 

beginning  of  1997  were  unsuccessful. 

According  to  this  e-mail,  between  September 
11  and  16,  1996  23,436  e-mail  messages  were 
missing  headers  (that  is,  ‘To,”  “Subject,”  etc.) 

and  10,138  messages  were  missing  text. _ 

Minutes  of  a  staff  meeting  involving  Jim  Wright, 

Laura  Crabtree,  Karl  Heissner,  and  others. 

Backup  of  EOP  Information  Systems  and 
Technology  Division  (IS&T)  servers  was 
discussed.  The  minutes  stated  that 
“[rjequirements  haven’t  been  documented  but 
approaches  are  being  developed.  An  inventory 
does  exist  of  what  does  need  to  be  backed  up.” 

The  minutes  also  question  what  the  retention 
period  should  be,  and  stated  that  all  types  of 
server  platforms  “are  being  reviewed:  VAX, 

client  server,  and  mainframe.” _ 

E-mail  to  Ada  Posey  and  others  regarding  El  022,  El  394, 

restricted  1997  appropriation  funding  and  that  Exhibit  9 

funds  were  needed  for  additional  e-mail  servers 

to  relieve  network  strain.  E-mail  records 

management  traffic  was  projected  to  increase 

90  percent  in  FY  1998  from  FY  1995’s  traffic 

rates.  The  e-mail  noted  that  EOP’s  migration 

from  Novell  Netware-based  network  to 

Microsoft  Windows  NT  began  in  May  1996.  It 

also  stated  that  the  “...client/server  backup 

system  used  to  preserve  data  integrity  and 

facilitate  the  archival  of  Presidential  and 

Federal  records  has  exceeded  its  life  cycle  [and 

can  only]  backup  50%  of  our  file  servers. 

However,  due  to  hardware  and  software 
failures,  less  than  30%  of  our  file  servers  have 
current  backups.  Currently,  the  only  copy  of 
data  that  is  not  backed  up  resides  on  each  file 
server’s  physical  hard  drives.  If  any  of  those 
hard  drives  fail,  data  will  be  lost.” 


Comments8 

Based  on  available 
documentation,  this 
problem  was  not  related 
to  the  Mail2  or  Letter  D 
malfunctions. 
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Malfunctions 


Reference8 

documents 

E6005  through 
E6009 


E2379  through 
E2380 


E7209  through 
E7211 


Exhibit  49 


Exhibit  10 


Date 

03/24/97 


07/17/97 


08/11/97 


01/30/98 


No  date 


Cross-reference 
to  other 
documents 
listed  in  this 

Description  of  event8  table8 

Minutes  of  a  staff  meeting  involving  Laura 
Crabtree,  Nell  Doering,  Karl  Heissner,  Jim 
Wright,  and  others.  The  minutes  discussed 
external  e-mail,  stating  “Laura  will  have  a  report 
on  the  problems  we’ve  been  having  for  Ada 

[Posey]  tomorrow.” _ 

E-mail  from  Bruce  Overton  to  Laura  Crabtree 
and  others  with  attached  e-mail  notice  to  all 
EOP  mail  users  regarding  implementation  of 
Notes  janitor  agent.  The  janitor  agent  was  set 
to  move  up  to  300  e-mail  messages  that  were 
greater  than  45  days  old  from  the  “inbox”  folder 
to  the  “trash”  folder  in  the  user’s  mail  file.  The 
messages  were  then  automatically  deleted  1 
week  later.  The  process  was  scheduled  to  run 
weekly  beginning  July  18,  1997.  Mr.  Overton 
asked  whether  the  records  management  team 

had  reviewed  the  janitor  policy. _ 

E-mail  to  Daniel  Barry,  Nellie  Doering,  Karl 
Heissner,  Laura  Crabtree,  Ada  Posey,  and 
others  containing  minutes  of  a  staff  meeting. 

The  minutes  state  that  “Lotus  Notes  staff  will  be 
[coming]  in  to  address  the  lost  mail  problem  (we 
need  a  ruling  from  [counsel]  as  to  issues  with 

mail  and  as  to  what  may  be  purchased).” _ 

A  document  created  by  Daniel  Barry  said  that  Exhibit  57 

he  noticed  a  potential  problem  with  the  capture 

of  e-mail  traffic  coming  into  the  EOP  Lotus 

Notes  system  from  the  Internet.  John  Spriggs, 

the  NG  e-mail  administrator,  was  asked  to  print 

out  the  firewall  log  to  determine  whether 

incoming  Internet  e-mail  messages  were 

received.  The  document  stated  “We  are  not 

sure  where  the  problem  lies  at  this  point.” _ 

A  document  containing  May  16  meeting  Exhibit  12 

minutes  discusses  a  problem  with  the 

Notes/ARMS  interface  scan  process  randomly 

skipping  e-mail  boxes  on  the  Mail2  server.  A 

random  sample  of  e-mail  boxes  was  analyzed 

and  it  was  discovered  that  e-mails  dating  back 

to  1996  were  not  subject  to  records 

management,  or  captured  by  the  ARMS 

records  management  database.  The  course  of 

action  was  to  further  assess  the  problem. 


Comments8 

Based  on  available 
documentation,  these 
problems  were  not  related 
to  the  Mail2  or  Letter  D 
malfunctions. 


A  9/29/00  response  to  our 
question  on  this  matter 
stated  that  “....  EOP  does 
not  believe  that  this 
‘janitor  agent’  was  ever 
implemented.”  In  its 
comments  on  a  draft  of 
this  report,  EOP  stated 
that "...  legal  and  records 
management  staff  never 
approved  use  of  the 

agent.” _ 

Based  on  available 
documentation,  this 
problem  was  not  related 
to  the  Mail2  or  Letter  D 
malfunctions. 


A  computer  directory 
location  and  date 
indicating  1/30/98  were 
handwritten  on  the 
document. 
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Exhibit  16 

06/01/98 

E-mail  from  Virginia  Apuzzo  to  all  EOP  staff 
requesting  that  users  delete  unneeded  e-mail 

E7202  through 

E7203 

on  e-mail  servers  or  save  e-mail  to  individual 
workstations  to  prevent  system  failure.  This 
message  was  repeated  and  re-sent  to  all  EOP 

_ staff  on  03/24/99  (Exhibit  16,  page  E0545). _ 

Exhibit  12  06/12/98  E-mail  from  Robert  Haas,  NG  contract  Exhibit  10  The  term  “design  flow” 

employee,  to  Betty  Lambuth,  NG  subcontractor,  may  be  a  typo  of  “design 

mentioning  “a  design  flow  in  the  Record  flaw”. 

Management  design  in  Notes  that  would  allow 
for  certain  types  of  mail  documents]  to  bypass 

_ record  management.” _ 

Exhibit  50  06/18/98  A  draft  document  entitled  Lotus  Notes  to  ARMS 

Interface  Anomaly  that  was  sent  from  the  fax 
machine  shared  by  the  Office  of  Administration 
(OA)  General  Counsel  and  the  OA  Director  to 
an  unknown  recipient.  The  draft  described  the 
Mail2  server  malfunction  and  stated  that  the 
root  of  the  problem  stems  from  a  discrepancy 
between  the  capitalization  of  the  letters  in  the 
name  of  the  affected  Lotus  Notes  server.  The 
problem  was  introduced  by  human  error  in 
capitalization  of  the  server  name  letters  in  the 
affected  users’  e-mail  accounts  as  “MAIL2” 
instead  of  “Mail2.”  The  scope  of  the  problem 

_ was  that  526  users  of  Mail2  were  affected. 

Exhibits  1  through  5  06/19/98  Memo  from  Virginia  M.  Apuzzo  to  John  D. 

Podesta,  former  Deputy  Chief  of  Staff,  entitled 
Technical  Anomaly  in  Automated  E-Mail 
Records  Management  System,  which  describes 
the  capitalization  of  the  server  name  letters  in 
the  affected  users’  e-mail  accounts  as  “MAIL2” 
instead  of  “Mail2”  as  the  cause  of  the  Mail2 
malfunction.  Exhibits  1  and  3  contained  a 
handwritten  note  saying,  “Chuck — I  sent  this 
memo  to  John  this  afternoon.  Ginny”  dated 
6/19.  Exhibits  4  and  5  contained  a  handwritten 
note  saying,  “Ginny  Please  ask  Mark  to  brief 

_ me  on  this.  Thanks,  John.” _ 

07/09/98  Draft  plan  dated  July  1 998  and  the  undated  Exhibit  1 9 
finalized  versions  of  the  same  plan  to  first 
process  data  records  from  Notes  e-mail  files  to 
the  VAX  subsystem  and  then  to  restore 
Notes/ARMS  interface  processing  on  Mail2. 

The  plan  was  to  first  turn  the  Notes/ARMS 
interface  monitor  task  off,  copy  each  affected 
user’s  existing  mail  file  (containing  the 
unrecorded  mail)  to  the  Notes/ARMS  interface 
_ server,  and  then  turn  the  Notes/ARMS  interface _ 


E0567  through 
E0569,  E0573 
through  E0578, 
E0589  through 
E0594 


Exhibits  1  through  5  were 
variations  of  the  same 
memo.  U.S.  District  Court 
testimony  by  Charles  Ruff 
on  08/28/00  confirmed 
“Chuck”  to  be  Charles 
Ruff,  “Ginny”  to  be 
Virginia  Apuzzo,  and 
“John”  to  be  John 
Podesta. 
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monitor  back  on  for  processing  of  unrecorded 
messages  into  the  ARMS  records  management 
database.  Once  transferred,  all  documents 
would  be  marked  as  recorded  and  the  syntax 
[capitalization  of  the  server  name  letters  in  the 
affected  users’  e-mail  accounts  as  “MAIL2” 
instead  of  “Mail2”]  of  the  server  name  for  the 
user’s  account  would  be  corrected.  Day-forward 
processing  would  resume  on  Mail2.  The  plan 
also  noted  that  a  “critical  issue”  existed  for  the 
processing  of  affected  records  from  Notes  to 
the  ARMS  records  management  database, 
which  resides  on  the  VAX  subsystem.  It  stated 
that  “preliminary  investigation  has  shown  that 
the  date  used  to  create  the  ARMS  header  files 
is  the  date  that  the  messages  are  written  to  the 
Records  Management  mail-in  database.  This 
date  doesn’t  reflect  the  true  sent  mail  date  in 
the  original  message.” 

Exhibit  19 

07/24/98 

E-mail  from  Daniel  Barry  to  James  Wright 
noting  under  Additional  activities  that  he 
continues  to  be  involved  in  discussions 
regarding  the  Mail2  problem  but  there  has  been 
no  movement  thus  far  in  correcting  the  problem 
or  getting  the  data  over  to  the  ARMS  records 
management  database.  The  plan  for  fixing  the 
problem  had  been  submitted  to  EOP  by  NG. 

Exhibit  22 

07/30/98 

E-mail  from  Daniel  Barry  to  James  Wright  with 
language  similar  to  Exhibit  19. 

Exhibit  23 

08/13/98 

E-mail  from  Daniel  Barry  to  James  Wright 
expressing  “concerns”  about  “the  mail2 
problem”  or  project  X.  He  stated  that  there  was 
no  movement  underway  to  fix  the  problem  and 
recover  the  lost  records  from  the  backup  tapes. 

Exhibit  24 

Exhibit  24 

08/13/98 

E-mail  from  James  Wright  to  Daniel  Barry 
responding  to  his  “concerns”  e-mail  saying  that 
there  has  been  some  movement  to  get  back  on 
track  and  NG  can  develop  a  plan  to  get  this 
effort  going.  “Certainly  the  Data  Center  and  the 
Records  Team  has  been  left  out  of  this  matter 
and  the  result  could  be  a  great  deal  of  work  put 
upon  us  later.”  This  exhibit  stated  that  Kathleen 
Gallant  informed  Mr.  Wright  that  a  Paulette 
[Cichon]  briefed  Jim  Welsh  of  NG  to  proceed 
with  developing  the  plan. 

Exhibit  23 
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Exhibit  40  08/13/98  E-mail  from  Kathleen  Gallant  to  James  Wright 

with  a  copy  to  Daniel  Barry.  It  stated  that  Jim 
Webster  was  taking  Betty  Lambuth’s  place  and 
a  meeting  was  held  authorizing  that  it  was  okay 
to  discuss  the  Mail2  project  in  detail  with  Mr. 

Webster.  The  e-mail  stated  that  “I  also  agree 
with  Tony  [Daniel  Barry]  about  the  new 
searches  that  will  have  to  be  done.  We  need 

_ direction  from  OA  counsel  on  that  front.” _ 

Exhibit  25  09/01/98  E-mail  from  Daniel  Barry  to  James  Webster, 

with  a  copy  to  James  Wright,  asking  for  a 
meeting  to  discuss  the  plan/approach  on  the 
Mail2  problem.  An  attached  e-mail  “Weekly” 
report  dated  9/4/98  (E2016)  says  that  Mr.  Barry 
had  a  discussion  with  Jim  Webster,  NG, 

“regarding  OVP  E-mail  and  records 
management.”  Mr.  Barry  referred  Mr.  Webster 
to  Kathleen  Gallant  for  a  decision  to  have  all 
OVP  users  subject  to  records  management 
through  the  ARMS  records  management 
database.  The  e-mail  noted  that  a  goods  and 
services  authorization  request  memo  was 
prepared  for  64GB  disks  for  the  Notes  Mail2 
recovery  project.  This  will  go  to  Financial 
Management  Division  (FMD)  for  signature  and 

_ then  make  its  way  to  the  OA  counsel. _ 

Exhibit  120  09/01/98,  E-mail  from  Daniel  Barry  to  James  Webster, 

09/15/98  NG,  requesting  meeting  with  him,  Sandra 
Golas,  Robert  Haas,  and  John  Spriggs  to 
discuss  the  Mail2  problem  and  the 
plan/approach  for  proceeding.  Webster 
forwarded  Barry’s  e-mail  to  Spriggs,  Golas, 

Robert  Haas,  and  Yiman  Salim,  saying  Barry 
wants  “to  set  up  a  meeting  to  discuss  our 
favorite  issue....  I  will  tell  Tony  [Daniel  Barry] 
that  I  will  have  to  delay  any  meeting  until  we 
resolve  some  internal  issues.”  On  page  E4015 
of  this  exhibit,  an  e-mail  dated  9/15/98  from 
Daniel  Barry  to  James  Webster  about  the  Mail2 
project  meeting  asks  about  Webster’s  plan  for 
“righting  the  wrong”  phase  of  this  project.  Barry 

_ wants  to  know  when  the  meeting  will  occur. _ 

E5874  09/10/98  E-mail  from  Mark  Bartholomew  to  Adam  E6020  through  The  bulk  e-mail  “package” 

Greenstone  [copy  to  Daniel  Barry,  Karl  E6022  was  captured  in  the 

Heissner,  Kathleen  Gallant]  regarding  ARMS  records 

processing  of  incoming  Internet  e-mail  to  the  management  database. 

President,  Vice  President,  and  the  First  Lady, 
referred  to  as  White  House  “principals.” 

_ Incoming  e-mail  to  these  principals  was _ 


Page  26 


GAO-Ol-446  EOP  E-mail  System 


Appendix  II:  Chronology  of  E-mail 

Malfunctions 

Cross-reference 

Reference8 

documents 

Date 

Description  of  event8 

to  other 
documents 
listed  in  this 
table8 

Comments8 

“packaged”  into  another  e-mail  to  the 
correspondence  office  for  each  respective 
principal.  These  “packaged”  e-mails,  referred  to 
as  “bulk  mail,”  were  not  subject  to  records 
management  in  the  ARMS  records 
management  database  message  by  message 
or  as  e-mail  to  the  principal,  but  rather  as  bulk 
e-mail  to  the  appropriate  representative  of  the 
principal. 

Exhibit  26 

09/10/98 

E-mail  from  Daniel  Barry  to  Kathleen  Gallant 
and  James  Wright  expressing  concern  about 
the  lack  of  movement  on  the  Mail2  problem. 

“We  have  known  about  this  problem  for  4 
months  now  and  not  a  single  record  [of  the 
affected  users]  has  been  passed  to 

ARMS. ..even  worse,  the  root  problem  has  not 
been  fixed.” 

Exhibit  64 

09/14/98 

Letter  from  Joseph  Lucente  noting  “that  in  late 
May  of  this  year,  a  dysfunction  in  the  EOP  e- 
mail  system  was  detected  by  an  employee...”  of 
NG. 

Exhibit  27 

09/25/98 

E-mail  from  Daniel  Barry  to  James  Wright 
stating  there  was  still  no  movement  in  the  Mail2 
problem  and  asking  what  his  role  was 
supposed  to  be  in  the  project.  Wright  responds 
the  same  day  saying  that  IS&T  needs  to  start 
by  providing  NG  with  direction. 

Exhibit  47 

09/25/98 

E-mail  from  Kathleen  Gallant  to  Daniel  Barry 
stating  that  the  Notes  Anomaly  Team  has  met 
and  come  up  with  a  strategy.  The  team  will 
back  up  Mail2  using  a  DLT  [digital  linear  tape] 
drive  that  can  verify  as  it  writes  and  execute 
“the  Notes  agent  changes  to  change  all  the 
[IDs]  looking  for  Mail2  to  MAIL2....”  A  second 
phase  deals  with  reconstruction  of  the 
nonrecords-managed  files  from  the  Mail2 
server.  She  said,  “Contracts  is  aware  of  the 

El  235  and  E1236 

Agents  are  filtering 
instructions  that  automate 
operations  on  documents 
in  a  Notes  database. 

whole  mess,  and  supports  the  creation  of  the 
IWO  [Internal  Work  Order]....” 

El 363  through 

El  375 

09/30/98 

A  NG  document  entitled  Final  Y2K  Status 

Report  for  OA/IS&T stated  that  the  software  for 
the  ARMS  records  management  database  was 
not  fully  documented  and  controlled,  and  that  it 
was  “born  very  quickly  in  1994  to  meet  an 
immediate  need  for  the  recording  of  message 
traffic.”  The  C-language  programs  in  the 
interface  were  not  Year  2000-compliant. 

E0766  and  E1015 
through  El  021 

Year  2000  readiness  of 
information  systems  was 
a  concurrent  effort  to 
repair  e-mail 
malfunctions. 
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Exhibit  28 

10/13/98 

E-mail  from  Daniel  Barry  to  Christina 

VanFossan  and  Joseph  Kouba  stating,  “Per  our 
meeting  last  week...  here  are  4  projects  that  1 
can  see  happening  in  the  near  term  that  are  not 
currently  budgeted  for.”  The  third  project  was 
listed  as  Project  X  (Mail2)  Reconstruction,  with 
an  estimated  cost  of  $250,000. 

E0766  through 
E0768 

11/06/98, 

11/12/98 

E-mail  created  by  Marvin  Miller  on  11/06/98 
and  revised  11/12/98  by  Yiman  Salim  regarding 
an  ARMS  IWO  feasibility  and  requirements 
study.  The  document  stated  that  Notes  and 

El  363  through 

El  375 

“Name  case  sensitivity” 
refers  to  the  capitalization 
of  the  server  name  letters 
in  the  affected  users’  e- 

Notes  Application  Programming  Interface  (API) 
version  4.6  software  was  used  in  the 

mail  accounts  as  “MAIL2” 
instead  of  “Mail2.” 

Notes/ARMS  interface.  To  perform 
maintenance  on  the  ARMS  software,  the  skill 
set  required  a  C++  software  engineer,  a  Notes 
developer  with  C++  and  VAX  knowledge,  and  a 
system  administrator  with  Windows  NT  and 

Lotus  Notes  skills.  The  tasks  listed  in  the  e- 
mails  included  Year  2000  code  modification 

In  its  comments  on  a  draft 
of  this  report,  EOP  stated 
that  the  Mail2  case 
sensitivity  was  not  related 
to  Year  2000  efforts 
addressed  by  the  “ARMS 
IWO.” 

and  implementation  of  calendaring  and 
scheduling  modifications.  However,  in  the 

1 1/12/00  revision,  Salim  removed  two  tasks 
from  Miller’s  list,  including  “Mail  server  name 
case  sensitivity.” 

Exhibit  30 

11/12/98 

E-mail  from  Daniel  Barry  to  DeVere  Patton 

Exhibits  31  and 

saying  that  NG  needs  technical  guidance  on 
several  items,  including  a  response  to  the  Mail2 
IWO,  which  had  not  yet  been  finalized. 

46 

Exhibit  31 

11/13/98 

E-mail  from  Daniel  Barry  to  James  Wright 

Exhibits  30  and 

mentioning  the  response  on  the  Mail2  IWO. 

46 

Exhibit  121 

11/16/98 

E-mail  from  Daniel  Barry  to  Sheryl  Hall,  John 
Spriggs,  and  Joseph  Vasta  requesting  a 
meeting  to  discuss  a  course  of  action  on 
moving  the  Mail2  reconstruction  project 
forward. 

Exhibit  103 

11/20/98 

E-mail  from  Daniel  Barry  stating  that  Karl 
Heissner  was  the  project  manager  for  phases 
two  and  three  of  the  Mail2  reconstruction  work. 
The  e-mail  mentioned  that  phase  one  was  the 
“....  fix  (Stop  the  bleeding)  on  the  mail  2 
server.”  Phases  two  and  three  were  not 
described  in  this  e-mail. 
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Exhibit  70 

12/02/98 

Weekly  Contracting  Officer’s  Technical 
Representative  (COTR)  meeting  minutes  for 

1 1/20/98.  NG  commented  that  the  Rough  Order 
of  Magnitude  (ROM)  “...could  be  plus  or  minus 
20  percent” 

Exhibits  63,  67, 

72,  73,  75,  and 

122 

Exhibit  72 

12/02/98 

NG  ROM  for  cost  of  IWO  was  $602,492. 

Exhibits  63,  67, 

70,  73,  75,  122, 
E1016 

Exhibit  122 

12/02/98 

E-mail  from  Daniel  Barry  to  Karl  Heissner 
stating  that  NG’s  ROM  did  not  include  the  cost 
to  do  the  restoration. 

Exhibits  63,  67, 

70,  72,  73,  and  75 

E5783 

12/04/98 

E-mail  from  Moe  Vela  to  “All  Staff”  requesting 
staff  to  delete  unnecessary  e-mail  because 
“Lotus  Notes  is  within  5  [percent]  of  maximum 
capacity  and  it  will  shut  down  at  that  time.” 

Exhibit  63 

12/11/98 

Memo  from  Joseph  Vasta  to  DeVere  Patton, 
mentioning,  in  part,  the  Mail2  E-mail 
Reconstruction.  Said  NG  would  evaluate  at 
least  one  tape. 

Exhibits  67,  70, 

72,  73,  75,  and 

122 

Exhibit  73 

12/18/98 

Weekly  COTR  meeting  minutes  for  12/16/98. 
“Government  emphasized  that  before  any  work 
be  undertaken,  it  is  imperative  that  a  tape 
inventory  be  done.”  NG  insisted  that  an 
inventory  in  the  absence  of  an  automated  tape 
management  system  would  be  very  labor- 
intensive  (over  1  hour  per  tape)  and  “would  far 
exceed  the  services  in  the  Base  Services  IWO.” 

Exhibits  63,  67, 

70,  72,  75,  and 

122 

Exhibit  67 

12/23/98 

Weekly  COTR  meeting  minutes  for  12/23/98. 

NG  reported  that  a  recent  tape  was  restored, 
and  the  universe  of  tapes  was  identified  as  788. 

Exhibits  63,  70, 

72,  73,  and  122 

Exhibit  32 

12/24/98 

E-mail  from  Daniel  Barry  to  Joseph  Kouba.  It 
lists  three  tiers  of  projects  related  to  the 
decisions  resultina  from  the  Armstrona  court 
case.  The  third  tier  includes  projects  that,  in  his 
opinion,  could  be  done  but  were  not  vital.  The 
Mail2  reconstruction  (Project  X)  was  listed  in 
the  third  tier,  showing  an  FY  1999  cost  of 
$650,000  and  an  FY  2000  cost  of  $1  million. 

The  Armstrona  court  held 
that  EOP  components 
must  capture  electronic 
records  subject  to  the 
Federal  Records  Act. 

Exhibit  129 

01/06/99 

Weekly  report  from  Daniel  Barry  in  which  he 
stressed  that  OA  Counsel  needed  to  be  in  the 
decision-making  process  regarding  projects 
that  should  move  forward,  such  as  Project  X. 

E6020  through 
E6022 

01/07/99 

Meeting  agenda  and  handwritten  meeting  notes 
regarding  “records  disposition  for  internet  e- 
mail.”  It  stated  that  for  e-mail  to  “...  ‘President 
@whitehouse.gov,’  all  emails  [were]  combined 
into  one  and  sent  to  [the  President’s  bulk  e-mail 

E5874 
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representative].”  It  was  packaged  and  sent  on 
to  the  ARMS  records  management  database 
where  it  was  retained  as  e-mail  to  the 

President’s  bulk  e-mail  representative. 

E5784 

01/13/99 

E-mail  from  Moe  Vela  to  “All  Staff”  requesting 
staff  to  delete  unnecessary  e-mail  and 
databases  in  Lotus  Notes  because  the  IS&T 
anticipated  running  out  of  disk  space  in  about  1 
hour. 

E0772 

02/01/99 

E-mail  from  Daniel  Barry  about  “the  effort 
underway  to  locate  and  document  the 
[Notes/ARMS]  interface  code  so  that  changes 
can  be  made.” 

E0964 

02/01/99 

E-mail  from  Daniel  Barry  to  Raul  Cavazos 
stating  that  “....  the  ARMS  processing 
requested  a  tape  from  March  1997  and  Ops 
[Operations]  was  unable  to  find  the  tape.” 
Concludes,  “there  needs  to  be  an  inventory  of 
the  ARMS  tapes  that  are  stored  in  the  boxes  in 
the  back  of  the  DC  [data  center],  to  make  it 
possible  to  retrieve  any  one  of  the  tapes.” 

Exhibit  75 

02/02/99 

Weekly  COTR  meeting  minutes  for  01/27/99. 

The  EOP  asked  about  NG’s  tape  inventory  and 
tape  tracking  methodology.  The  EOP  requested 
a  tape  inventory  (that  included  tape  ID,  volume, 
dataset  ID,  etc.)  from  October  1997,  the  point 
when  NG  was  awarded  contract. 

Exhibits  63,  67, 

70,  72,  73,  and 

122 

Exhibit  81 

02/05/99 

Memo  from  Karl  Heissner  explaining  the  Mail2 
server  problem.  He  stated  that  “the  problem, 
that  some  e-mail  messages  were  not  records 
managed  properly,  was  discovered  in  February 
1998,  and  Mr.  Barry  immediately  notified  the 
Lotus  Notes  Group  (NG-Spriggs/Golas)  as  well 
as  IS&T  Management  of  the  problem.” 

Exhibits  82  and 

83 

Exhibit  43 

03/12/99 

E-mail  from  Nellie  Doering  to  Dorothy  Cleal, 
copy  to  Daniel  Barry,  suggesting  changes  to 

Mr.  Barry’s  proposal.  Ms.  Doering  stated  that 

Mr.  Barry  agreed  to  ensure  the  “Server  2”  tapes 
were  properly  inventoried  and  documented  by 
NG. 

Exhibit  37 

03/18/99 

E-mail  from  Daniel  Barry  to  DeVere  Patton,  Karl 
Heissner,  and  Nellie  Doering  about  the  Mail2 
records  management  problem.  Barry  stated 
that:  “It  has  come  to  my  attention  that  when  the 
‘bleeding’  was  stopped  on  MAIL2  in  November 
1998,  ALL  the  bleeding  may  NOT  have  been 
stopped.  1  have  spoken  with  John  Spriggs  and 
it  appears  as  though  at  least  one  account, 
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MILLENNIUM,  may  still  have  the  problem.  I 
believe  NG  should  be  instructed  to  investigate 
and  report  back  on  exactly  what  the  situation  is 

_ with  regard  to  the  MAIL2  problem.” _ 

Exhibit  41  03/18/99  A  series  of  e-mails  noting  that  the  Mail2 

problem  was  not  allowing  EOP  staff  to  recycle 
backup  tapes.  Nellie  Doering  stated  that  the 
tapes  need  to  be  inventoried  by  NG  and  “.... 

‘secured’  so  that  they  are  easily  identified  as 
the  Server  2  Backup  Tapes  that  were  not 
[subject  to  records  management.]”  DeVere 
Patton  stated  that  “...it  is  something  that  NG 
should  be  doing  under  the  base  contract  [in 
response  to  Dorothy  deal’s  inquiry  regarding 

_ contract  scope], _ 

Exhibit  42  03/18/99  E-mail  from  Joseph  Kouba  to  Daniel  Barry  and 

others  stating,  “looks  like  MAIL  2  reconstruction 
is  back  on  hold  until  some  additional 

_ confirmation  is  received.” _ 

Exhibit  124  03/19/99  E-mail  from  Karl  Heissner  to  Nellie  Doering, 

Daniel  Barry,  and  Mark  Bartholomew.  Meeting 
minutes  to  discuss  the  Mail2  reconstruction.  It 
stated  that  the  project  was  on  hold  awaiting 

_ approval  of  the  OA  counsel. _ 

E0812  03/24/99  E-mail  from  Michael  Ritter  to  Joseph  Vasta 

regarding  tape  inventory.  It  stated  that  “there 
are  between  700  and  800  tapes  [in]  mostly  8- 
mm  [millimeter]  format  and  do  not  have  a 
‘header’  catalogue  (meaning  that  the  entire 
tape  must  be  read  to  develop  a  directory  list).” 

Also,  “there  is  a  short  period  of  a  few  months 
where  we  do  not  have  backup  tapes  due  to 
changes  in  the  governments  requirements.”  Mr. 

Ritter  estimates  that  under  “perfect  conditions” 
it  would  take  1 .5  to  3  hours  per  tape  to  read  the 
tape  and  print  the  catalogue  information; 
however,  he  stated,  the  tapes  were  old  and 
were  produced  under  various  hardware  and 
software  configurations.  He  stated  that  “it  is 
expected  that  there  will  be  tapes  that  cannot 
even  be  read  with  the  current  hardware  on 

_ site.” _ 

03/25/99  NG  memo  from  Joe  Vasta  to  DeVere  Patton 

regarding  Mail2  backup  tapes.  It  stated  that  NG 
will  secure  Mail2  backups  for  October  1996 
through  November  1998  and  provide  minimal 
inventory  information.  The  memo  identified  that 
during  this  period,  NG  was  instructed  by  “the 

_ government”  to  reuse  backup  tapes  causing _ 


E1255  through 
El  256 


Reference8 
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Exhibit  117 


Date  Description  of  event8 

some  to  be  overwritten.  The  memo  stated  that 
a  report  detailing  a  description  of  the  system 
failure  was  beyond  the  scope  of  the  base 

_ contract. _ 

04/01/99  E-mail  from  Michael  Ritter  to  Yiman  Salim, 

Robert  Haas,  John  Spriggs,  and  Sandra  Golas 
asking  them  a  series  of  questions  about  when 
the  Mail2  server  problem  started  and  when  it 
was  resolved.  He  requested  responses  from 
each  by  4/2/99. 


Cross-reference 
to  other 
documents 
listed  in  this 
table8 


E0833  and  E0834 


Comments8 


El  194 


Exhibit  66 


04/01/99 


04/02/99 


Exhibit  71 


04/02/99 


Exhibit  44 


04/09/99 


E-mail  from  Benjamin  Kirby  to  Adam 
Greenstone,  Daniel  Barry,  others,  stating  the 
millennium  account  was  not  subject  to  records 

management. _ 

E-mail  from  Yiman  Salim  to  Michael  Ritter 
about  the  Mail2  issue.  She  stated  that  NG 
personnel  became  aware  of  the  Mail2  problem 
on  June  12,  1998,  and  that  the  cause  of  the 
problem,  writing  Mail2  as  MAIL2,  was  identified 
a  couple  of  days  later  by  John  Spriggs.  She 
also  noted  that  the  problem  was  inherent  in  the 
ARMS  process  because  it  “is  currently 
enforcing  server  name  case  sensitivity.”  The 
message  also  stated  that  in  December  1998 
NG  initiated  a  task  to  baseline  the  Notes/ARMS 
interface  program.  Completion  was  scheduled 
for  April/May  1999,  after  which  programming 

changes  could  be  made  as  needed. _ 

E-mail  from  Michael  Ritter  to  Joseph  Vasta. 

This  e-mail  explains  the  Mail2  server  problem  in 
detail.  It  also  mentions  that  NG  staff  have 
identified  a  similar  problem  on  MAIL1  that  was 
also  caused  by  human  error.  The  message 
stated  that  an  employee  of  the  contractor  PRC 
designed  a  program  to  migrate  user  accounts 
to  the  Notes  server,  and  this  program  entered 
the  users  with  the  improper  server  name 

syntax. _ 

Meeting  notice  from  Karl  Heissner  about 
records  management  of  all  Mail2  and  MAIL1 
server  accounts.  This  notice  stated  that  Mail2 
and  MAIL1  servers  still  contain 
“unrecordsmanaged”  accounts,  preventing  the 
recycling  of  backup  tapes.  The  objective  was  to 
make  necessary  corrections  by  April  18. 


Exhibit  117 
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Exhibit  125  04/09/99  Series  of  e-mails  describing  Letter  D  Exhibits  13,  79, 

malfunction.  Marvin  Miller  e-mails  to  Robert  documents 
Whiteman  and  Yiman  Salim  that  the  Letter  D  E321 8,  E321 9 
malfunction  was  caused  in  1 1/98  when  NG  was  through  E3221 , 
correcting  a  Lotus  Notes  buffer  overflow  system  and  E3222 
failure.  The  user  accounts  (sorted  by  first  through  E3229 

initial/last  name)  were  distributed  among  five 
Notes/ARMS  interface  e-mail  server  views,  so 
that  all  26  letters  of  the  alphabet  were  assigned 
to  one  of  the  5  views.  “The  letter  ‘D’  was 
inadvertently  omitted  and  the  letter  ‘J’  was 
added  twice.”  This  caused  all  users  on  all  e- 
mail  servers  whose  first  names  began  with  D  to 

_ not  be  subject  to  records  management. _ 

E321 9  through  04/15/99  Memo  from  Robert  Whiteman  (NG)  to  Dorothy  Exhibit  79, 

E3221  Cleal  describing  events  regarding  the  Letter  D  documents 

malfunction.  It  stated  that  a  Notes/ARMS  E321 8,  and 

interface  e-mail  server  system  failure  occurred  E3222  through 

in  November  1998,  which  was  caused  by  a  E3229 

configuration  error.  In  correcting  the  failure,  NG 

reconfigured  the  Notes/ARMS  interface  views 

on  Lotus  Notes  e-mail  servers  and 

“inadvertently  omitted”  the  letter  D  from  the 

views,  causing  incoming  non-Notes  e-mail  for 

users  with  first  names  beginning  with  D  not  to 

be  subject  to  records  management.  According 

to  the  memo,  the  Lotus  Notes  team  reported 

this  to  NG  management  on  04/09/99.  Mr. 

Whiteman  recommends  not  recycling  tapes 
until  the  problem  was  resolved  and  to  include 
the  saved  tapes  in  the  ongoing  e-mail  recovery 

_ effort. _ 

E3218  04/15/99  Memo  from  Dorothy  Cleal,  through  Michael  Exhibits  13  and 

Lyle,  to  Mark  Lindsay  regarding  the  Letter  D.  79;  documents 
The  memo  stated  that  1 91  users  were  affected  E321 8,  E321 9 
and  who  have  not  had  e-mail  records-managed  through  E3221 , 
since  November  1998.  Cautions  that  “...records  and  E3222 
management  problem  may  not  be  limited  to  this  through  E3229 

_ particular  error.” _ 

Exhibit  79,  E3222  04/15/99  Cover  memo  from  Eric  Ritter  to  Dorothy  Cleal  Exhibits  13,  125; 

with  attached  listing  that  shows  the  191  users  documents 
impacted  by  the  Letter  D  malfunction — 54  on  E321 8,  E321 9 
MAIL1 ,  77  on  Mail2,  29  on  MAIL3,  30  on  through  3221 ,  and 

MAIL5,  and  1  on  RDS1.  E3222  through 

E3229 


Exhibit  99  04/20/99  This  e-mail  from  Joseph  Kouba  stated  that  a 

“legal  determination  [was  needed]  that 
reconstruction  is  required  by  the  court  case  and 
_ that  this  is  a  legitimate  use  of  Armstrong  funds.” 
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E7265  through 
E7268 

04/28/99 

E-mail  from  Robert  Whiteman  to  Marvin  Miller, 
Robert  Haas,  Yiman  Salim  and  others 
requesting  review  of  Lotus  Notes  self 
evaluation  for  10/98  to  04/99.  It  stated  that 

Notes  staff  began  analysis  phase  of  the 
Notes/ARMS  interface  task.  During  this 
analysis,  staff  identified  a  records  management 
problem  (Letter  D  malfunction)  and  were 
researching  viable  solutions. 

El  235  and  E1236 

05/17/99 

Report  explaining  e-mail  audit  agents  that  run 
on  Notes  servers  and  search  for  unrecorded 
mail  and  incorrect  mail  templates.  It 
recommends  running  audit  agents  weekly  and 
providing  logs  to  Mr.  Barry.  It  also  recommends 
resuming  tape  recycling.  It  stated  that  “it  would 
not  be  possible  to  recycle  any  back-up  tapes 
that  were  created  prior  to  the  (managed  date) 
completion  of  the  correction  and  verification  of 
the  system...”. 

Exhibit  47 

Exhibit  38 

05/21/99 

E-mail  from  Michael  Ritter  to  Albert  Leister 
regaining  the  Mail2  Statement  of  Work  (SOW). 
“This  is  actually  from  September  ’98.  Still  no 
action  on. ..or  response  from. ..the  gov’t.” 

Exhibit  118 

05/21/99 

E-mail  from  Albert  Leister  Jr.  to  Dorothy  Cleal 
stating  that  the  “...backup  tapes  are  in  the  DC 
(data  center].” 

E0793 

05/21/99 

E-mail  from  Marvin  Miller  showing  a  schedule 
to  run  various  Notes  agents.  One  scheduled  for 
05/25/99  was  to  “fix  letter  ‘D’  and  letter  case 
problem.  Records  will  be  managed  from  this 
point  on.” 

E4483 

06/04/99 

E-mail  from  Albert  Leister  to  Dorothy  Cleal 
reporting  that  the  Notes  audit  agents  indicate  all 
documents  requiring  records  management 
were  managed  properly  and  the  agents  run 
weekly  “....  to  insure  no  new  errors  surface.” 

The  e-mail  noted  that  audit  logs  were  given  to 
Karl  Heissner  and  requests  personal  direction 
to  resume  tape  recycling. 

E0798 

06/09/99 

E-mail  from  Yiman  Salim  stating  that  “...the 
[Notes/ARMS  interface]  views,  in  the  Name  and 
Address  Book,  need  to  be  modified  to  make  the 
selection  criteria  non-case  sensitive.”  The 
Notes/ARMS  interface  scanning  process  had 
skipped  a  user  account  (that  was  not  in  use) 
because  the  name  started  with  lower  case.  “A 
CMC  [configuration  management  change]  has 
been  submitted  to  implement  this  modification.” 

Page  34 


GAO-Ol-446  EOP  E-mail  System 


Appendix  II:  Chronology  of  E-mail 
Malfunctions 


Reference8 

documents 

Date 

Description  of  event8 

Cross-reference 
to  other 
documents 
listed  in  this 

table8  Comments8 

Exhibit  6 

01/13/00 

Briefing  materials  for  1/19/00  meeting  with  the 
National  Archives  and  Records  Administration 
regarding  e-mail  issues.  It  mentions  Mail2  and 
Letter  D  configuration  issues.  It  stated  that 

Mail2  affected  526  users  and  Letter  D  affected 
about  200  users. 

Exhibit  130 

01/21/00 

Chronology  of  Mail2  server  events  and  impact, 
labeled  “Tony”  in  upper  right-hand  corner.  It 
discusses  the  problem  and  plan  of  action.  It 
also  identified  the  magnitude  of  e-mails  being 
researched  as  encompassing  850  tapes,  and 
that  some  tapes  contain  classified  data.  The 
plan  of  action  says  that  Enterprise  Computing 
Solutions  Technology,  Incorporated  (ECS),  an 

8a  firm,  has  prior  experience  and  was  currently 
on-site  for  the  Armstrong  effort. 

E4337  through 

02/01/00 

E-mail  from  Daniel  Barry  to  Michael  Sullivan 

E5978  through 

E4338 

with  comments  on  an  attached  draft  request  for 
proposals  (RFP)  for  NG  to  bid  on  recovery  of 
lost  e-mails.  Barry  comments  that  a 
subcontractor  to  Information  Management 
Consultants,  Incorporated  (IMC),  had 
developed  the  original  Notes/ARMS  interface 
code  in  1996.  The  e-mail  stated  that  the 
Notes/ARMS  interface  transitioned  to  NG  in 
1997/1998.  The  system  broke  in  late  1998 
because  it  could  not  handle  increased  user 
population  when  migrating  from  All-in-1  [the 
previous  e-mail  system]  to  Notes,  causing  the 
Letter  D  problem.  The  RFP  requests  a  proposal 
from  NG  by  02/15/00,  and  stated  that  NG 
caused  the  problem  and  that  “...corrective 
action  should  be  at  no-cost  to  the  Government.” 
Barry  asks  if  it  was  “fair  and  equitable”  to  hold 
NG  accountable  for  fixing  bugs  in  the  software 
developed  by  the  IMC  subcontractor. 

E5979 

El  393 

02/25/00 

Handwritten  notes  stating  that  some  tapes 
included  in  the  tape  recovery  effort  were 
classified  as  confidential  or  secret. 

El  394 

02/25/00 

Handwritten  notes  of  an  interview  about  “Bob’s 
-  Mail  issue,”  which  state  that  backup  problems 
were  experienced  due  to  going  from  a  File 
Allocation  Table  (FAT)  system  to  a  Windows 

NT  File  System  (NTFS).  An  accompanying 
timeline  shows  the  backup  problem  existed 
from  the  beginning  of  the  Mail2  problem  (10/96) 
until  about  11/97. 

Exhibit  9,  El  022, 
and  E6552 
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E3237 

No  date 

Document  entitled  Tape  Reconstruction 
Procedure  Outline  stated  that  850  tapes  were 
involved  in  the  recovery  effort. 

E4349  through 
E4354 

02/29/00 

Ontrack  Computer  Evidence  Services’ 
Statement  of  Work  and  Proposal  for  Services 
proposes  a  cost  of  $436,000  plus  expenses 
and  55  days  to  copy  and  catalog  800  tapes  at 
the  EOP,  which  would  require  acquisition  of  1 1 
computers.  It  would  cost  $348,000  and  55  days 
to  perform  the  work  at  Ontrack’s  facility,  which 
was  fully  equipped.  Pricing  for  data  recovery 
varied  depending  on  four  categories  of  optional 
services. 

E3247  through 
E3248 

03/01/00 

E-mails  between  John  Spriggs,  Albert  Leister, 
Terrence  Misich,  and  William  Burkey  regarding 
backup  tapes.  Magnitude  of  tapes  revised  by 
Burkey  to  3,391 ,  including  283  “miscellaneous 
tapes.”  Spriggs  stated  that  “we  really  do  not 
have  any  good  evidence  of  what  backup 
[device]  was  actually  backing  up  Mail2  on  any 
given  date.”  Leister  identified  three  boxes  of 
tapes  containing  1 ,006  8-mm  tapes  and  51 
DLTtapes  as  definite  to  search  and  identified 
another  three  boxes  containing  503  8-mm 
tapes  as  “possible,”  for  a  total  of  1 ,560  tapes. 
The  tapes  consist  of  4-mm,  8-mm,  and 

DLTtape  media  spanning  intermittent  periods 
from  1/96  to  6/99.  The  tapes  reportedly 
contained  backup  of  Office  of  Management  and 
Budget  (OMB),  White  House  Office,  and  other 
EOP  agency  servers,  including  Mail2. 

Exhibit  46 

No  date 

Timeline  for  ARMS-related  activities 
(Development  Tasks,  Problem  Discovery  and 
Problem  Fixing).  It  lists  the  Mail2  problem  as 
discovered  on  6/12/98  and  fixed  on  11/22/98. 
The  Letter  “D”  was  problem  discovered  on 

4/9/99  and  fixed  on  6/1/99. 

Exhibit  9 

No  date 

Draft  Talking  Points  on  the  Mail2  Server 
Anomaly  stated  that  incoming  e-mails  exist  on 
backup  tapes,  “...except  for  a  small  window  of 
time  from  8/96 — 11/97  when,  due  to  a  glitch  in 
the  backup  software,  the  mail  files  may  not 
have  been  backed  up  on  a  consistent  basis.” 

El  022,  El  394, 
and  E6552 

El  040  through 

El  041 

03/10/00 

Memo  from  Terrence  Misich  to  Jack  Young, 
General  Counsel/OA.  It  stated  that  he  “visually 

El  01 5  through 

El  021 
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scanned  14  boxes  of  magnetic  media, 
containing  3,391  magnetic  tapes  of  various 
types”  and  that  a  number  of  tapes  do  not  have 
external  labels  and  some  may  have  erroneous 
external  labels.  Mr.  Misich  recommends  the 
performance  of  the  first  phase  of  the  Tape 
Restoration  Plan. 


Exhibit  13 


03/14/00 


Cover  note  from  Yiman  Salim  with  attached 
audit  log  files  from  May  8  and  9,  1999,  for  EOP 
mail  servers  Mail2  and  MAIL1  showing 
unrecorded  documents.  According  to  the  logs, 
in  May  1999,  both  Mail2  and  MAIL1  had  four 
user  accounts  each  with  server  name  syntax 
errors  causing  unrecorded  mail  dating  back  to 
1997.  The  logs  also  show  126  users  affected  by 
the  “Letter  D”  malfunction  (70  on  Mail2  and  56 
on  MAIL1). 


Exhibits  37,  44,  Logs  show  that  about  5 
71, 79  and  125.  months  after  the  Mail2  fix 
was  implemented,  Mail2 
still  had  four  user 
accounts  with  syntax 
errors  and  unrecorded 
mail  dating  back  to  1997- 
1998.  Also,  MAIL1  had 
four  user  accounts  with 
syntax  errors  in  this  same 


time  frame. 


El  01 5  through  03/1 5/00  A  document  entitled  Draft  Tape  Restoration  Exhibit  72,  El  363 

El  021  Plan  revised  to  include  new  data  from  ECS.  It  through  El  375, 

stated  that  3,391  tapes  were  involved  in  the  and  E0776 

recovery  effort  and  attributes  the  Letter  D 

malfunction  to  a  “...software  failure.”  It  also 

attributes  the  Mail2  malfunction  to  case 

sensitivity.  It  stated  that  “work  associated  with 

this  data  recovery  began  almost  as  soon  as  the 

problem  was  discovered.”  The  draft  plan 

mentioned  that  the  work  was  not  begun  in 

October  1998  when  the  SOW  was  developed 

for  NG  because  of  Year  2000  efforts,  so  the 

recovery  was  postponed  to  March  2000.  In 

February  2000,  OA  “...began  exploring 

alternatives,”  and  received  an  estimate  from 

one  company  of  $436,000  and  55  days  to 

extract  mail  from  850  tapes.  The  draft  plan 

extended  these  estimates  to  $1 ,744,000  and 

220  days  for  3,391  tapes.  The  draft  plan  also 

stated  that  another  vendor,  familiar  with  ARMS, 

_ estimated  $3,286,596  and  1 .7  years. _ 

El  022  No  date  Hand-drawn  timeline  of  e-mail  malfunctions.  El  394,  E6552,  In  its  comments  on  a  draft 

Shows  October  1 996  to  October  1 997  “backup  and  Exhibit  9  of  this  report,  EOP  stated 
tapes  missing,  FAT  to  NSF  file. ..Fixed.”  August  that  “....  tapes  were  not 

1 998  “ARMS  research  to  fix  Mail2  -  discover  recycled  from  June  of 

they  cannot.”  November  20,  1 998,  Mail2  stop  1 998  until  June  of  1 999.” 

bleed,  tape  recycling  starts  November  21 .  April 
9,  1999,  D  problem  identified,  stopped  tape 
recycle.  May  21,  1999,  “stopped  D-bleed”. 
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Exhibit  57 

03/17/00 

Fax  from  Beth  Nolan,  White  House  Counsel,  to 

Exhibit  50 

The  Honorable  Dan  Burton,  stated,  “In  January 
1998,  Daniel  Barry.... noticed  a  possible 
anomaly  within  ARMS. ..that  some  incoming  e- 
mails  might  be  missing. ...Mr.  Barry  notified  his 
superiors  and  documented  his  finding. ..The  full 
extent  of  the  error  causing  the  anomaly  Mr. 

Barry  noted  was  not  discovered  until  June 
1998,  when  NG  employees 
discovered.... certain  incoming  e-mail 
messages  that  were  coded  as 
“unrecorded”.... The  contractor  notified  IS&T 
personnel. ...By  the  fall  of  1998,  NG  technical 
personnel  working  with  IS&T  staff  discovered 
that  the  problem  was  due  to  miscoding  Mail2’ 
as  ‘MAIL2.’  They  further  determined  that  the 
miscoding  affected  526  ARMS-managed 
accounts....  By  November  1998,  the  NG  and 
IS&T  personnel  had  corrected  the  problem....” 
prospectively 

“...incoming  e-mail  to  ARMS-managed 
accounts  with  the  first  names  beginning  with 
the  letter  ‘D’  had  not  been  recorded  by  ARMS 
since  November  1998.  It  appears  that  this  error 
remained  undetected  until  April  1999....” 

Approximately  200  accounts  within  EOP  were 
affected. 

“In  the  course  of  gathering  these  preliminary 
facts  concerning  these  configuration  errors,  we 
were  informed  this  week  that  e-mails  on  the 
server  of  the  Office  of  the  Vice  President  (OVP) 

have  not  been  fully  managed  by  ARMS." _ 

“References  are  to  the  House  Committee  on  Government  Reform  exhibits  used  at  the  March  2000 

hearings;  additional  documents  provided  to  the  Committee  are  identified  by  an  “Exxxx”  scheme. 

Bracketed  items  within  the  “Description  of  Event”  column  and  the  “Comments”  column  have  been 

added  by  us  for  readability  and  clarification. 


Page  38 


GAO-Ol-446  EOP  E-mail  System 


Appendix  III:  Key  Officials  Involved  in 
Primary  Mail2  and  Letter  D  Events 


Executive  Office  of 
the  President  Federal 
Staff,  October  1996 
Through  June  1999: 

Paulette  Cichon,  former  Deputy  Director,  Information  Management,  OA 

Dorothy  Cleal,  former  Associate  Director,  OA/IS&T 

Laura  Crabtree,  former  Desktop  Systems  Branch  Chief,  OA/IS&T 

Nellie  Doering,  Records  Management,  OA/IS&T 

Kathleen  Gallant,  former  Associate  Director,  OA/IS&T 

Sheryl  Hall,  former  Computer  Specialist,  OA/IS&T 

Karl  Heissner,  Systems  Integration  and  Development  Branch  Chief, 
OA/IS&T 


Virginia  Apuzzo,  former  Assistant  to  the  President  for  Management  and 
Administration 

Daniel  A.  “Tony”  Barry,  Computer  Specialist,  Office  of  Administration 
(OA)/Information  Systems  and  Technology  Division  (IS&T) 

Mark  Bartholomew,  Web  Development,  OA/IS&T 


Joseph  Kouba,  Budget  Analyst,  Financial  Management  Division 

Mark  Lindsay,  Assistant  to  the  President  for  Management  and 
Administration,  former  General  Counsel,  OA,  and  former  Director,  OA 

Michael  Lyle,  Director,  OA,  and  former  General  Counsel,  OA 

Cheryl  Mills,  former  Deputy  Counsel  to  the  President 

Terrence  Misich,  Chief  Warrant  Officer  5,  U.S.  Army,  on  detail  from  White 
House  Communications  Agency  to  OA/IS&T,  Tape  Restoration  Project 
Manager 

Beth  Nolan,  Counsel  to  the  President 
Bruce  Overton,  Deputy  General  Counsel,  OA 

DeVere  Patton,  Contracting  Officer’s  Technical  Representative  (COTR)  for 
the  Northrop  Grumman  Contract 
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John  Podesta,  Chief  of  Staff 

Ada  Posey,  former  Director,  OA 

Charles  Ruff,  former  Counsel  to  the  President 

Michael  Sullivan,  Deputy  Associate  Director  for  General  Services  Division, 
OA 

Christina  VanFossan,  Director,  Financial  Management  Division 

James  Wright,  former  COTR  for  the  Northrop  Grumman  contract,  and 
Data  Center  Branch  Chief,  OA/IS&T 


Northrop  Grumman 
Contract  Staff  From 
Contract  Award  in 
October  1997  Through 
June  1999: 


Sandra  Golas,  VAX  System  Administrator 

Robert  Haas,  Lotus  Notes  E-mail  System  Administrator,  Lotus  Notes  team 
Steve  Hawkins,  former  Project  Manager 

Betty  Lambuth,  CEXEC  (a  subcontractor  to  Northrop  Grumman),  former 
Lotus  Notes  team  manager 

Joseph  Lucente,  Director  of  Contracts  and  Subcontracts 
Yiman  Salim,  Lotus  Notes  Developer,  Lotus  Notes  team 
John  Spriggs,  Senior  Engineer 
Joseph  Vasta,  former  Program  Manager 
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Appendix  IV:  Time  Line  of  Key  Events  in  the 
Mail2  and  Letter  D  Malfunctions 


Tape  “backup  gap” 


♦I 


I - 

Normal  tape  recycling  occurs 
8/96  1 1/96  2/97 


08/96 
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server 

created 


10/96 
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Office  of 
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President 


(EOP)  adds 
user 

accounts  to 
Mai]2  with 


MA1L2 
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•  EOP 


migrates 
from  Novell 
Netware  to 
Windows 
NT 

Operating 
system — 
backup 
problems 
result 


*\ 


i - \ - 1 - h 


10/97 

Tape 

backup 

problem 

from 

10/96 

fixed 


01/98 
Daniel 
Barry 
records 
incident 
report  of 
anomaly 
involving 
external 
e-mail 


05/98 

Northrop 

Grumman 

staff 

identify 

Mail2 

anomaly 


06/98 

«  Cause  of 
Mail2 

documented 

6/18/98 

•  EOP  senior 
staff  notified 
of  Mail2 
malfunction 
6/19/98 

•  Tape 
recycling 
stopped 
6/19/98 


11/98 

•  Mai!2 
malfunction 
fixed  11/20/98 

•  Tape  recycling 
resumed 
11/21/98 

•  System  failure 
due  to  buffer 
overflow  in 
views  on  mail 
servers-NG 
staff 

implement  fix 
and 

inadvertently 
omit  users 
with  first 
name 
beginning 
with  D  from 
ARMS 
management 


I - ► 

Tapes  recycled 


2/99  5/99  8/99 


i  i  y 


04/99 
Letter  D 
malfunction 
discovered 
by  NG  staff 


05-06/99 

•  Letter  D 
malfunction 
fixed 
05/21/99 

•  Tape 
recycling 
resumes 
06/07/99 


Source:  GAO  compilation  of  documents  provided  by  EOP  to  the  House  Committee  on  Government 
Reform. 
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the  Tape  Restoration  Project 


Cost  element 

Description 

Estimated  cost 

ECS  contract: 

Restoration  of  e-mail  messages  not  recorded  in  ARMS 

Phase  1 

Task  1 :  Develop  a  detailed  project  plan 

$  836,600 

Task  2:  Define,  configure,  acquire,  and  install  required  equipment 

862,300 

Task  3:  Develop  a  quality  assurance  and  quality  control  program  for  the  project 

217,600 

Task  4:  Develop,  test,  and  refine  tape  recovery  software 

742,600 

Task  5:  Test  and  validate  recovery  plan  and  software 

128,400 

Phase  II 

Task  6:  Receive  and  inventory  all  tapes 

118,000 

Task  7:  Make  two  copies  of  each  tape 

364,000 

Task  8:  Place  original  tapes  in  secure  custody 

31,000 

Task  9:  Scan  duplicated  tapes  into  message  recovery  system 

337,000 

Phase  III 

Task  10:  Process  e-mail  into  searchable  database 

92,400 

Task  1 1 :  Remove  duplicate  e-mail  messages  and  prepare  for  transfer  to  ARMS 

93,000 

Task  12:  Place  duplicate  e-mail  messages  into  a  searchable  database 

105,400 

Task  13:  Process  classified  tapes  as  above 

208,000 

Task  14:  Deliver  final  products 

80,000 

General  and  administrative  expenses  and  other  direct  costs 

909,529 

ECS  total: 

$5,125,829 

Vistronix  contract: 

Independent  validation  and  verification  of  tape  restoration 

Phase  1 

Tasks  1  through  5  (see  above) 

$  465,483 

Phase  II 

Tasks  6  through  9  (see  above) 

1,078,233 

Phase  III 

Tasks  10  through  14  (see  above) 

450,103 

Vistronix  total: 

$1,993,819 

Other  costs: 

Other  restoration  costs 

AAC  Associates  contract 

Advice  on  recovery  of  OVP  e-mails 

$900,000 

Vendor  not  determined 

Forensic  analysis  and  recovery  of  broken  or  unreadable  tapes 

2,092,400 

Vendor  not  specified 

Additional  hardware,  software  licenses,  and  12,000  tapes  for  two  sets  of  tape  copies 

498,480 

Not  applicable 

Project  overrun  contingency 

1,098,891 

Other  costs  totals 

$4,589,771 

Total 

$11,709,419 

Source:  Tape  restoration  project  cost  estimate  report  prepared  by  Vistronix,  Incorporated,  for  the 
EOP  in  July  2000. 
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the  Former  Vice  President 


Note:  GAO  comments 
supplementing  those  in 
the  report  text  appear  at 
the  end  of  this  appendix. 


OFFICE  OF  THE  FORMER  VICE  PRESIDENT 
THE  HONORABLE  AL  GORE 


April  11,  2001 

Mr.  Joel  C.  Willemssen 
Managing  Director 
Information  Technology  Issues 
U.S.  General  Accounting  Office 
441  G  Street,  N.W. 

Washington,  D.C.  20548 

Dear  Mr.  Willemssen: 

Enclosed  please  find  the  comments  of  the  Office  of  Former  Vice  President 
A1  Gore  on  the  General  Accounting  Office  (GAO)  draft  report  entitled  Electronic 
Records:  Clinton  Administration 's  Management  of Executive  Office  of  the  President  E- 
mail  System  (hereinafter  “the  Report”). 

I  was  not  principally  responsible  for  the  information  flow  related  to 
GAO’s  inquiry  into  EOP  e-mail  management  during  the  former  administration. 

Therefore,  I  have  compiled  the  following  comments  in  consultation  with  members  of  the 
former  administration  who  were  responsible  for  preparation  of  responses  to  the  GAO 
inquiry.  The  comments  below  arc  divided  into  two  sections:  “General  Commentary”  and 
“Line  by  Line  Commentary.” 


General  Commentary 

While  the  Report  chronicles  several  miseommunications  among  EOP 
components  related  to  e-mail  system  support  and  records  management,  it  fails  to  state 
explicitly  that  there  is  absolutely  no  evidence  of  any  attempt  by  OVP  or  EOP  personnel 
to  deliberately  fail  to  preserve  any  documentary  material  or  inappropriately  withhold 
information  from  any  investigative  body.  Considering  the  nature  of  some  of  the 
allegations  that  have  been  made  in  relation  to  this  matter,  the  interests  of  fairness  dictate 
that  this  fact  should  be  made  explicit. 

Another  point  that  does  not  come  through  in  the  current  draft  of  the  Report 
relates  to  the  ARMS  system.  ARMS  did  not  exist  before  the  EOP  contracted  to  have  it 
created  early  in  the  former  administration.  While  EOP  and  OVP  personnel  may  have 
come  to  know  that  ARMS  was  searchable,  ARMS  was  not  created  for  this  purpose. 
ARMS  was  created  to  capture  and  preserve  e-mail  as  a  method  of  compliance  with  the 
Federal  Records  Act  (FRA)  for  EOP  components  governed  by  it.  ARMS  was  not  used  as 
a  subpoena  and  document  request  compliance  tool  until  sometime  much  later  in  the 
administration.  The  ARMS  system  was  sui  generis  and,  therefore,  it  would  be  difficult  to 
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the  Former  Vice  President 


Now  on  p.  1 . 
See  comment  1 . 


Now  on  p.  1 . 
See  comment  2. 


Now  on  p.  2. 
See  comment  3. 


Now  on  p.  2. 
See  comment  4. 

Now  on  p.  2. 
See  comment  5. 

Now  on  p.  2. 
See  comment  6. 


measure  technical  problems  integrating  new  EOP  components  and  software  systems  into 
ARMS  against  “best  practices.”  Best  practices,  as  they  relate  to  the  unique  ARMS 
system,  continue  to  evolve.  In  fact,  the  former  administration’s  experiences  with  ARMS, 
some  of  which  are  chronicled  in  this  Report,  should  serve  to  help  develop  the  best 
practices  as  they  relate  to  ARMS  technology  and  its  intersection  with  the  Presidential 
Records  Act  (PRA)  and  the  FRA. 


Line  by  Line  Commentary 

The  following  is  a  list  of  specific  textual  changes  urged  by  the  Office  of 
the  Former  Vice  President  to  be  incorporated  into  the  final  Report.  Each  comment  in  this 
section  begins  with  a  location  of  the  text  in  question  in  a  parenthetical.  Bold  typeface 
denotes  recommended  textual  insertions  and  strikethrough  typeface  denotes 
recommended  textual  deletions. 

(First  p.  2, 1 1, 2nd  sentence)  To  be  accurate,  the  sentence  should  read:  “. .  .require  the 
EOP  to  preserve  certain  official  records,  including  e-mail  qualifying  as  presidential  or 
vice  presidential  records.” 

(First  p.  2,  1,  3rd  sentence)  The  PRA  does  not  require  that  presidential  records  are 

“available  for  use  in  the  course  of  official  duties.”  Rather,  the  PRA  requires  that 
presidential  records  be  preserved  and  transferred  to  the  custody  of  the  Archivist  of  the 
United  States  upon  the  termination  of  the  President  and  Vice  President’s  term  in  office. 
See  44  U.S.C.  Chapter  22. 

(Second  p.  2,  carryover  ^|)  GAO  requested  to  physically  inspect  randomly  sampled  paper 
files  within  the  OVP  on  the  week  of  December  11,  2000.  This  request  either 
misunderstood  that  the  OVP  did  not  have  a  central  filing  system  for  printed  out  e-mails  or 
it  was  an  overly  broad  request  to  review  OVP  files  without  regard  to  any  specific  subject 
matter.  These  OVP  objections  were  raised  with  GAO  through  then-OA  personnel  at  the 
time  yet  GAO  made  no  effort  to  refine  its  request  or  explain  why  its  original  request 
served  a  properly  circumscribed  purpose. 

(Second  p.  2,  1st  full  *[)  As  discussed  in  the  “General  Commentary”  section,  while 
ARMS  is  searchable,  it  was  not  created  for  that  purpose. 

(Second  p.  2,  2nd  full  f)  As  discussed  in  the  “General  Commentary”  section,  “best 
practices,”  as  they  relate  to  the  unique  ARMS  system,  continue  to  evolve. 

(Second  p.  2,  2nd  full  K,  3rd  Sentence)  To  be  accurate,  the  sentence  should  read:  “The 
OVP  discontinued  this  approach  at  some  point  after  May  1993— in  the  mistaken  belief 
that  the  e-mail  records  were  being  archived  in  ARMS  and  that  another  IS&T,  the 
division  of  the  Office  of  Administration  that  performed  that  function  for  other  EOP 
components,  was  managing  the  backup  tapes.” 
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Now  on  p.  2. 

See  comment  7. 

Now  on  pp.  2  to  3. 
See  comment  8. 

See  comment  9. 

Now  on  p.  4. 

See  comment  10. 

Now  on  p.  5. 

See  comment  1 1 . 

Now  on  pp.  5  to  6. 
See  comment  12. 


Now  on  p.  7. 

See  comment  13. 


Now  on  p.  9. 

See  comment  14. 

Now  on  p.  10. 
See  comment  15. 


Now  on  p.  10. 
See  comment  16. 


(Second  p.  2, 2nd  full  4th  sentence)  To  be  accurate,  the  sentence  should  read:  “The 
OVP  could  not  demonstrate  that  all  e-mail  records. 

(Second  p.  2,  2nd  full  ti,  5th  sentence)  To  be  accurate,  the  sentence  should  read:  “As  a 
result,  about  600  system  backup  tapes  needed  restoration  to  determine  if  any  non- 
archived  e-mail  records  existed  on  the  tapes.” 

(p.  3,  2nd  full  bullet  point)  To  be  accurate,  the  sentence  should  read:  “Legal  and 
Congressional  scrutiny  of  the  e-mail  malfunctions. . .” 

(p.  3,  1st  full  1|  in  “Background”  section,  1st  sentence)  To  be  accurate,  the  sentence  should 
read:  “ . . .  adequately  record  their  official  acts,  maintain  all  such  certain  official  records, 
and  transfer  custody  of  such  records. 

(p.  4,  footnote  1,  second  sentence)  We  do  not  agree  with  the  assertion  that  “system 
backup  tapes  are  not  to  be  used  for  recordkeeping  purposes.”  Upon  what  authority  does 
GAO  base  this  assertion? 

(p.  4,  1st  full  1|  following  bullet  points)  This  paragraph  does  not  make  it  clear  that  the 
FRA  does  not  apply  to  the  Office  of  the  Vice  President.  Certain  official  records 
generated  by  the  Office  of  the  Vice  President  are  subject  to  the  PRA.  See  44  U.S.C.  § 
2207.  Therefore,  as  it  relates  to  the  assertion  in  footnote  2,  OVP  vice  presidential  records 
were  transferred  to  the  custody  of  IS&T  for  the  limited  purposes  of  e-mail  system 
administration  and  records  management  and  those  records  did  not  become  federal  records 
at  any  time.  For  example,  vice  presidential  e-mail  records  in  the  custody  of  IS&T  for 
these  limited  purposes  were  not  subject  to  the  Freedom  of  Information  Act  requests, 
whereas  OA  federal  records  were  so  subject. 

(p.  5,  4th  full  f  in  “History  Surrounding. . .”  section,  5lh  sentence)  This  sentence  fails  to 
recognize  that  ARMS  was  not  created  for  the  purpose  of  subpoena  compliance. 

Therefore,  to  be  accurate,  the  sentence  should  read:  “The  ARMS  database  maintained  e- 
mail  records  in  a  searchable  format  which  came  to  support  facilitate  EOP’s  responses  to 
document  requests  and  subpoenas  sometime  much  later  in  the  administration. 

(p.  7,  2nd  full  |  in  “The  OVP  Did  Not. . .”  section,  1st  sentence)  To  be  accurate,  the 
sentence  should  read:  “During  a  portion  of  the  administration,  the  OVP  was 
responsible...” 

(p.  7,  2nd  full  H  in  “The  OVP  Did  Not.. .”  section,  3rd  sentence)  To  be  accurate,  the 
sentence  should  read:  “However,  the  OVP  and  IS&T  did  not  effectively  communicate 
its  requirements  which  office  was  responsible  for  management  of  its  OVP  e-mail 
records  and  the  OVP  then  assumed  that  records  management  responsibility  also 
transitioned  to  the  Office  of  Administration  as  was  true  for  other  EOP  components.” 

(p.  7,  2nd  full  f  in  “The  OVP  Did  Not. . .”  section,  4rd  sentence)  To  be  accurate,  the 
sentence  should  read:  “According  to  the  BOP  Likewise,  the  Office  of  Administration. . .” 
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Now  on  p.  10. 
See  comment  17. 


Now  on  p.  10. 
See  comment  18. 


Now  on  p.  10. 
See  comment  19. 


Now  on  p.  10. 
See  comment  20. 

Now  on  p.  10. 
See  comment  21 . 

Now  on  p.  11. 
See  comment  22. 


Now  on  p.  11. 
See  comment  28. 


Now  on  p.  14. 
See  comment  24. 


Now  on  p.  16. 
See  comment  25. 


(pp.  7-8,  carryover  sentence)  To  be  accurate,  the  sentence  should  read:  “This  mistaken 
belief  was  based  on  the  precedent  that  ARMS  managed  e-mail  records  for  all  other  EOF 
components  since  its  implementation  in  1994,  on  the  fact  that  searches  of  ARMS 
produced  OVP  e-mail  records,  and  on  the  observation  that  some  outgoing  OVP  e- 
mails  were  in  fact  being  ARMS  managed.” 

(p.  8,  in  between  the  2nd  and  3rd  full  sentences)  This  paragraph  would  be  more  accurate  if 
the  following  sentence  were  added:  “These  records,  however,  are  being  restored  as 
part  of  the  Tape  Restoration  Project.” 

(p.  8, 3rd  full  sentence)  GAO  requested  to  physically  inspect  randomly  sampled  paper 
files  within  the  OVP  on  the  week  of  December  11, 2000.  This  request  either 
misunderstood  that  the  OVP  did  not  have  a  central  filing  system  for  printed  out  e-mails  or 
it  was  an  overly  broad  request  to  review  OVP  files  without  regard  to  any  specific  subject 
matter.  These  OVP  objections  were  raised  with  GAO  through  then-OA  personnel  at  the 
time  yet  GAO  made  no  effort  to  refine  its  request  or  explain  why  its  original  request 
served  a  properly  circumscribed  purpose. 

(p.  8,  1st  full  K,  1st  sentence)  To  be  accurate,  the  sentence  should  read:  “Although  the 
OVP  did  not  continually  retain  paper  copies  of  some  e-mail  records  not  captured  by 
ARMS  in  accordance  with  EOP  policy. . .” 

(p.  8,  1st  full  2nd  sentence)  To  be  accurate,  the  sentence  should  read:  “According  to  the 
EOP,  however,  because  these  OVP  backup  tapes  were  retained  for  system  recovery  and 
not  records  archival  only,  they  were  not  in  a  text-searchable  format.” 

(p.  8,  2nd  full  %  last  sentence)  To  be  accurate,  the  sentence  should  read:  “However, 

because  the  OVP  and  OA  IS&T  miscommunicated  during  the  transfer  of  operation 
and  maintenance  of  the  OVP  e-mail  system  did  not  ensure  the  preservation  of  e-mail 
records,  in-paper  electronic-form,  presidential  records  may  have  been  irretrievably  lost.” 

(p.  8,  footnote  9,  2nd  to  last  sentence)  To  be  accurate,  the  sentence  should  read:  “Some 
Senate  OVP  staff  used. . .”  The  OVP  correspondence  office,  located  in  the  Senate,  used  a 
different  system,  however  the  OVP  legislative  affairs  office,  also  located  in  the  Senate, 
used  the  same  system  as  the  rest  of  the  OVP. 

(p.  11,  1st  full  %  2nd  sentence)  To  be  accurate,  the  sentence  should  read:  “Such  a 
monitoring  program  should  include  evaluations  of  the  creation  identification  of 
presidential  and  vice  presidential  records,  maintenance  and  use ...” 

(p.  12,  last  full  •[)  This  paragraph  does  not  adequately  reflect  the  rationale  EOP  gave  for 
not  releasing  copies  of  those  30  letters.  As  a  matter  of  policy,  where  possible  the  EOP 
did  not  cross-pollinate  its  communications  with  various  investigative  bodies  as  a  means 
of  maintaining  the  confidentiality  of  the  legitimate  interests  and  investigative  priorities  of 
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Now  on  p.  16. 
See  comment  26. 


those  respective  investigative  bodies.  This  policy  was  communicated  to  GAO  at  the  time 
by  then-EOP  staff. 

(p.  1 3,  Conclusion)  As  discussed  in  the  “General  Commentary”  section  above,  the 
Report  makes  no  mention  of  the  fact  that  there  is  no  evidence  whatsoever  of  any  intent 
not  to  preserve  presidential  records  or  to  inappropriately  withhold  responsive  documents 
from  investigative  bodies.  Considering  the  Report’s  criticism  of  EOP  management 
practices,  such  an  explicit  statement  is  required  in  the  interests  of  fairness. 


*  *  * 

Please  do  not  hesitate  to  contact  me  with  additional  draft  versions  or  if  you 
would  like  to  discuss  the  contents  of  these  comments  further.  I  can  be  reached  at  the 
transition  office  directly  at  703.235.0409. 

Thank  you  very  much. 


Sincerely, 


Andrew  M.  Wright,  Esq. 

Counsel  to  the  Former  Vice  President 
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The  following  are  GAO’s  comments  on  the  office  of  the  former  Vice 
President’s  letter,  dated  April  11,  2001. 


GAO  Comments 


We  revised  the  report  to  include  the  word  “certain,”  but  did  not 
incorporate  the  remaining  text  as  suggested  by  the  Office  of  the  former 
Vice  President.  The  “certain”  records  that  the  report  states  are 
“deemed  official  government  records”  are  those  incoming  e-mail 
messages  not  captured  by  the  Automated  Records  Management 
System  (ARMS)  due  to  the  malfunctions.  These  records  consisted  of 
both  presidential/vice  presidential  and  federal  records. 


2.  We  revised  the  report  to  reflect  the  technical  clarification  provided. 


3.  We  submitted  written  questions  to  the  Office  of  the  Vice  President 
(OVP),  through  the  Executive  Office  of  the  President  (EOP),  on 
September  14,  2000,  in  which  we  asked  if,  and  how,  the  OVP 
implemented  the  portion  of  the  1997  EOP  policy  which  stated  that  staff 
“[m]aintain  Presidential  Records  in  organized  files.”  In  its  November  14 
response,  the  OVP  stated  that  OVP  staff  implemented  the  requirements 
of  the  statute  by  maintaining  presidential  records  “in  organized  files.” 

In  follow-up,  we  submitted  another  set  of  written  questions  to  the  OVP, 
through  the  EOP,  on  November  30.  At  this  time,  our  request  was  to 
“[p]lease  provide  a  list  of  all  OVP  staff  who  maintain  the  organized  files 
of  hard  copy  OVP  records  as  mentioned  in  the  EOP  response.  We 
would  like  to  physically  observe  the  files  of  randomly  selected  OVP 
staff  and  would  like  to  arrange  for  this  observation  during  the  week  of 
December  11,  2000.”  Our  purpose  in  reviewing  the  files  of  selected 
staff  was  to  respond  to  the  House  Committee  on  Government  Reform’s 
request  that  we  determine  whether  OVP  implemented  adequate 
practices  in  accordance  with  applicable  criteria  for  management  of  e- 
mail  records.  The  intent  of  our  request  was  to  confirm  the  existence  of 
e-mail  records  in  such  files  and  not  to  examine  the  text  of  the  records 
contained  therein,  as  explained  verbally  to  the  EOP  Office  of 
Administration  General  Counsel  on  numerous  occasions  during 
December  2000  and  January  2001.  We  were  not  provided  a  contact 
within  OVP  with  whom  to  discuss  the  request.  The  OVP  did  not 
respond  to  our  request  in  writing  until  6:49  p.m.  on  January  19,  2001,  at 
which  time  OVP’s  response  was  that  the  OVP  did  not  maintain  a 
centralized  filing  system.  In  its  response,  OVP  may  have  confused  the 
word  “organized”  in  our  request  with  “centralized.”  Several  hours  after 
we  were  provided  OVP’s  response,  the  administration  had  transitioned 
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and  the  records  were  in  the  process  of  transfer  to  the  National 
Archives  and  Records  Administration. 

4.  A  key  feature  of  an  electronic  recordkeeping  system  is  that  records  be 
retrievable.  To  facilitate  retrieval  of  records,  the  recordkeeping  system 
must  index  records  and  contain  fields  that  can  be  searched. 

5.  Best  practices  for  management  of  records  currently  exist  in  the  form 
of  National  Archives  and  Records  Administration  guidance  contained 
in  title  36  of  the  Code  of  Federal  Regulations  (CFR).  Even  though  the 
majority  of  this  guidance  is  prescriptive  only  for  federal  records,  it 
does  not  explicitly  preclude  application  of  these  practices  in  the 
management  of  presidential/vice  presidential  records.  Although  the 
evolution  of  ARMS  continues,  the  stability  of  generally  accepted 
records  management  best  practices  establishes  a  control  foundation 
for  any  system,  whether  manual  or  automated. 

6.  We  revised  the  report  to  identify  “another  EOP  component”  as  EOP’s 
Office  of  Administration. 

7.  We  revised  the  report  to  reflect  the  technical  clarification  provided. 

8.  We  revised  the  report  to  reflect  the  technical  clarification  provided. 

9.  The  report  was  revised  to  reflect  that  external  authorities  other  than 
those  of  law  enforcement  scrutinized  the  e-mail  malfunctions. 

10.  We  revised  the  report  to  incorporate  the  technical  clarification 
provided. 

11.  Title  36  CFR,  subpart  1234.24,  Standards  for  managing  electronic  mail 
records,  promulgates  this  standard  (36  CFR  1234.24(c)). 

12.  The  report  provides  a  description  of  common  federal  standards  for 
management  and  control  of  federal  programs  and  systems.  We  did  not 
imply  that  the  application  of  such  controls  alters  in  any  way  the 
determination  of  whether  a  record  is  presidential/vice  presidential  or 
federal.  The  inclusion  of  this  discussion  was  to  emphasize  that  federal 
programs,  such  as  records  management  programs,  and  systems,  such 
as  ARMS  and  the  e-mail  system,  are  subject  to  application  of  certain 
management  controls.  These  controls  ensure  the  effectiveness  of 
programs  as  well  as  ensure  that  adequate  integrity,  confidentiality,  and 
availability  of  electronic  information  resources  are  maintained.  Both 
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the  Presidential  Records  Act  (PRA)  and  the  Federal  Records  Act 
(FRA)  require  the  implementation  of  management  controls. 

13.  The  new  information  provided  in  the  Office  of  the  former  Vice 
President’s  comments  on  the  draft  has  been  incorporated  into  the 
report. 

14.  We  revised  the  report  to  reflect  the  technical  clarification  provided. 

15.  Under  the  PRA,  the  Vice  President  (and  thus  OVP),  was  responsible  for 
management  of  the  Vice  President’s  records.  From  the  beginning  of  the 
administration,  it  retained  the  primary  duty  to  implement  records 
management  controls,  and  thus,  to  either  modify  its  records 
management  practice  to  accommodate  the  system  transition  or  to 
ensure  that  this  responsibility  was  assigned  to  another  entity. 

16.  We  attribute  this  statement  to  EOP  because  it  was  taken  from  the 
White  House  memo  to  us  on  January  19,  2001,  which  stated  that  it 
provided  EOP’s  responses  to  our  questions  submitted  on  November  30, 
2000. 

17.  The  new  information  provided  by  the  Office  of  the  former  Vice 
President  has  been  incorporated  into  the  report. 

18.  The  new  information  provided  by  the  Office  of  the  former  Vice 
President  has  been  incorporated  into  the  report. 

19.  See  our  response  to  comment  3  above. 

20.  In  its  January  19,  2001,  response  to  our  written  questions,  the  EOP 
stated  that  at  some  point  between  issuance  of  the  May  1993  EOP 
policy  and  1998,  OVP  ceased  requiring  the  retention  of  paper  copies  of 
e-mail  records.  We  were  provided  no  evidence,  or  allowed  to 
physically  confirm,  that  some  printed  e-mail  records  were  retained. 
Also,  the  EOP  policy  did  not  explicitly  require  that  records  be  captured 
by  ARMS,  as  indicated  by  the  comment. 

21.  We  revised  the  report  to  reflect  the  technical  clarification  provided. 

22.  As  discussed  in  our  response  to  comment  15  above,  the  OVP  had  a 
duty  to  ensure  continual  preservation  of  its  vice  presidential  records, 
including  ensuring  that  this  responsibility  effectively  transferred  to  the 
Office  of  Administration. 
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23.  We  revised  the  report  to  reflect  the  technical  clarification  provided. 

24.  The  characteristics  described  in  the  report  are  based  on  National 
Archives  and  Records  Administration  standards  as  presented  in  title  36 
CFR.  Creation  of  records  is  the  manual  or  automated  process  during 
which  a  document  is  produced  and  is  determined  to  qualify  as  a 
record.  Identification  is  a  part  of  this  process  in  which  there  is 
recognition  that  a  document  is  a  record  and  a  determination  is  made  as 
to  the  type  of  record.  The  characteristics  described  pertain  to  the 
monitoring  of  both  presidential/vice  presidential  and  federal  records. 

25.  We  agree  with  the  Office  of  the  former  Vice  President’s  assessment; 
however,  the  only  rationale  that  was  provided  to  us,  either  orally  or  in 
writing,  was  that  the  letters  were  “deserving  of  respect  and  deference.” 
We  revised  the  report  to  incorporate  the  additional  rationale  provided 
in  its  comments  to  the  draft  report. 

26.  The  Committee  requested  that  we  assess  the  adequacy  of  management 
controls  over  EOP’s  records  management  system,  particularly  as  they 
relate  to  the  numerous  e-mail  system  malfunctions  experienced  by 
EOP  since  1996.  That  assessment  entailed  the  examination  of 
management  policies,  programs,  and  practices  as  they  related  to 
effective  records  management  controls  and  to  the  requirements  of  the 
PRA  and  FRA.  Our  review  was  not  directed  at  assessing  the  intent  of 
individuals  responsible  for  preserving  records  and  providing 
responsive  records  to  investigative  bodies.  As  such,  we  can  offer  no 
conclusions  on  this  matter. 


(310.307) 
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The  first  copy  of  each  GAO  report  is  free.  Additional  copies  of  reports  are 
$2  each.  A  check  or  money  order  should  be  made  out  to  the 
Superintendent  of  Documents.  VISA  and  MasterCard  credit  cards  are  also 
accepted. 


Orders  for  100  or  more  copies  to  be  mailed  to  a  single  address  are 
discounted  25  percent. 

Orders  by  mail: 

U.S.  General  Accounting  Office 
P.O.  Box  37050 
Washington,  DC  20013 

Orders  by  visiting: 

Room  1100 

700  4lh  St.,  NW  (corner  of  4,h  and  G  Sts.  NW) 

Washington,  DC  20013 

Orders  by  phone: 

(202)  512-6000 
fax:  (202)  512-6061 
TDD  (202)  512-2537 

Each  day,  GAO  issues  a  list  of  newly  available  reports  and  testimony.  To 
receive  facsimile  copies  of  the  daily  list  or  any  list  from  the  past  30  days, 
please  call  (202)  512-6000  using  a  touchtone  phone.  A  recorded  menu  will 
provide  information  on  how  to  obtain  these  lists. 

Orders  by  Internet 

For  information  on  how  to  access  GAO  reports  on  the  Internet,  send  an  e- 
mail  message  with  “info”  in  the  body  to: 


lnfo@www.gao.gov 


or  visit  GAO’s  World  Wide  Web  home  page  at: 


http://www.gao.gov 


To  Report  Fraud, 
Waste,  and  Abuse  in 
Federal  Programs 


Contact  one: 


Web  site:  pttp://www.gao.gov/fraudnet/fraudnet.htm 


E-mail:  fraudnet@gao.gov 

1-800-424-5454  (automated  answering  system) 
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